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INTRODUCTION 


ItACKflKOUND: 

During  the  past  two  years,  significant  organization  change  has 
occurred  within  the  Montana  Department  of  Institutions  (the  Depart- 
ment) .   A  primary  change  has  been  increased  centralization  of 
Department  management  and  administrative  functions.   For  example, 
Montana  House  Bill  699  mandated  consolidation  of  all  functions  of 
State  Government  dealing  with  Addictive  Diseases  into  the  Depart- 
ment.  This  consolidation  brought  together  under  the  Addictive 
Diseases  Bureau  (ADB) ,  four  existing  organization  units  dealing 
with  drug  abuse  and  alcoholism  treatment.   They  were: 

-Single  State  Agency  for  Drug  Abuse  Treatment  Coordination 
-Southwestern  Montana  Drug  Treatment  Program  ^ 

-Single  State  Alcohol  Authority 
-Montana  Alcohol  Service  Center 

Further,  an  Adaptive  Services  Division  (ASD)  was  created  within  the 
Department  with  a  single  administrator  responsible  for  the  new  ADB, 
the  Mental  Health  Bureau  (MHB)  and  all  state  institutions  for  the 
retarded,  mentally  ill  and  elderly.   This  reorganization  brought 
the  two  major  service  components  of  the  Department  (institutional 
care  and  community-based  care)  under  one  organization  unit. 

At  the  time  of  the  reorganization,  the  scope  of  ASD  responsibilities 
for  community-based  services  included  approximately  31  mental 
health  centers  or  satellites,  15  outpatient  or  residential  drug 
abuse  treatment  programs  and  22  primary  providers  of  alcoholism 
treatment  services.   Authorized  Federal  and  State  combined  expendi- 
tures for  FY  1977  were  approximately  5.8  million  dollars. 

The  single  most  important  responsibility  of  the  ASD  is  to  provide 
overall  direction  and  monitoring  of  these  community-based  programs 
to  help  ensure  statewide  program  compliance  with  Federal  and  State 
standards.   This  responsibility  is  clearly  mandated  by  Montana 
House  Bill  289  which,  in  part,  requires  biannual  review  of  com- 
munity-based programs  to  which  state  funds  have  been  allocated.   As 
the  number  and  diversity  of  local  treatment  programs  have  increased, 
the  requirements  for  program  accountability  have  become  more  complex. 
In  the  past,  there  existed  numerous  program  evaluation  methods  and 
formats  which  are  not  satisfactory  for  current  Department  needs  for 
the  following  reasons: 

-Evaluation  criteria  have  not  been  sufficiently  responsive  to 

State  requirements. 
-Program  evaluations  have  not  been  performed  on  a  regular  or 

timely  basis. 
-Evaluation  criteria  for  mental  health,  drug  abuse  treatment 

and  alcoholism  treatment  programs  have  not  been  uniform  in 

content  or  approach. 


-Methods  and  procedures  for  conducting  program  evaluations  were 
poorly  documented  resulting  in  lack  of  objectivity  and  in- 
completeness of  findings  and  suggested  corrective  action. 

To  help  correct  these  weaknesses  and  to  carry  out  the  responsi- 
bility for  monitoring  and  evaluation,  the  ASD  has  developed  this 
handbook  for  use  by  state  and  local  agency  personnel  to  achieve  the 
following  major  purposes. 

PURPOSE  OF  THE  PROGRAM  EVALUATION  PROCESS: 

The  purpose  of  the  program  evaluation  process  as  outlined  in  this 
handbook  includes  primary  and  secondary  objectives: 

Primary  objectives  are: 

-To  improve  program  performance  and  client  service  levels. 
-To  help  ensure  statewide  compliance  with  Federal  and  State 

standards. 
-To  help  achieve  efficiency  and  economy  in  program  operations. 

Secondary  objectives  are: 

-To  assist  in  determination  of  program  funding  levels  and 

approval  of  contracts . 
-To  determine  areas  of  needed  technical  assistance  and  training. 
-To  gather  data  for  development  of  additional  state  standards 

for  program  performance. 

The  guidelines  and  procedures  presented  in  this  handbook  are  in- 
tended to  help  achieve  these  objectives.   However,  the  procedures 
included  cannot  supplant  the  evaluator's  judgment  as  to  the  optimal 
approach  to  follow  in  verifying  the  existence  or  quality  of  all 
required  operations.   In  many  instances,  additional  work  may  be 
necessary  to  fully  understand  and  assess  program  activities. 

SUMMARY  OF  HANDBOOK  CONTENTS: 


The  handbook  is  organized  around  three  primary  sections.   Section  I  is 
a  general  introduction  to  the  background  and  purpose  of  the  program 
evaluation  process.   Section  II  outlines  the  general  steps  to  be  taken 
in  conducting  the  on-site  program  evaluation  and  is  intended  for  use  by 
ASD  personnel.   This  section  presents  an  outline  of  important  management 
considerations  in  achieving  efficient  and  effective  evaluation  of 
numerous  community-based  programs.   Section  III  includes  a  summary  of 
Federal  and  State  standards  and  detailed  evaluation  procedures  to  determine 
whether  standards  are  met  for  each  of  four  evaluation  areas: 

-Organization  and  Program  Management 
-Personnel  Management  and  Staff  Development 
-Financial  Management 
-Client  Treatment  and  Census 


This  section  is  intended  for  use  not  only  by  ASD  program  evaluation 
personnel  but  also  by  all  community-based  program  management  staff 
for  conducting  self-evaluation.   Attached  to  the  handbook  is  an 
appendix  which  includes  a  program  evaluation  checklist  for  use  in 
summarizing  findings  and  recommendations  resulting  from  the  evalua- 
tion process. 


GENERAL  GUIDELINES  AND  PROCEDURES 


OVERVIEW  OF  AN  ON-SITE  PROGRAM  EVALUATION: 


The  on-site  evaluation  provides  an  opportunity  for  the  program 
evaluator  to  assess  the  controls  and  management  systems  within  a 
program.   For  the  purposes  of  this  handbook,  a  program  is  defined 
as  a  community-based  service  provider  of  mental  health,  drug  abuse 
treatment  or  alcoholism  treatment  services.   An  assessment  of 
management  systems  enables  the  evaluator  to  determine  whether  a 
program  is  operating  in  a  manner  consistent  with  Federal  and  State 
program  standards  and  accountability  requirements.   Program  manage- 
ment is  defined  in  terms  of  four  basic  components: 

-Organization  and  Program  Management  Systems 
-Personnel  and  Staff  Development  Systems 
-Financial  Management  Systems 
-Client  Treatment  Systems 

The  organization  system  consists  of  methods  by  which  the  program 
manages  itself.   The  personnel  and  staff  development  system  in- 
clude the  policies  and  procedures  used  to  attract,  maintain  and 
develop  staff.   The  financial  system  consists  of  guidelines  and 
procedures  which  govern  accounting,  budgeting  and  equipment  pro- 
curement activities.   The  client  treatment  system  consists  of 
procedures,  documentation  and  reporting  which  relate  to  client 
treatment  activities. 

Each  of  these  systems  may  be  evaluated  using  one  or  all  of  the 
following  methods  of  program  assessment: 

-Review  of  written  policies  and  procedures 

-Review  of  program  files 

-Interviews  with  staff  and  management  personnel 

Evaluation  results  should  be  completely  documented  in  evaluator 
notes  indicating  the  names  of  staff  interviewed,  date  of  interview 
and  documents  reviewed.   In  addition,  summary  comments  should  be 
included  indicating  the  rationale  used  in  determining  whether 
the  program  meets  standards.   Then,  results  are  summarized  on 
the  program  evaluation  checklist  displayed  in  the  appendix  of  the 
handbook.   This  checklist  is  designed  for  the  most  comprehensive 
evaluations  and  is  organized  according  to  the  four  basic  systems  of 
program  management.   It  is  important  to  point  out  here  that  the 
checklist  provides  for  only  a  summary  of  findings  and  should  not  be 
used  until  after  evaluation  procedures  have  been  performed  and 
documented. 

Three  types  of  site  visits  are  defined  -  comprehensive,  interim  and 
follow-up.   Each  has  a  different  focus  and  requires  a  different 
level  of  manpower,  but  all  follow  the  same  basic  phases  of  performance. 
Each  type  is  defined  in  detail  later  in  this  section. 


The  program  evaluation  consists  of  the  following  phases: 

-Scheduling 

-Pre-planning 

-Entry  interview 

-Work  planning 

-Conducting  the  review 

-Exit  interview 

-Preparation  and  distribution  of  the  final  report 

-Conducting  the  debriefing  interviews  with  the  program  director 

A  discussion  of  the  three  types  of  evaluation  and  each  phase  of  the 
evaluation  process  makes  up  the  remainder  of  this  section  of  the 
handbook. 

B.    TYPES  OF  EVALUATION: 

In  order  to  provide  evaluation  flexibility,  three  types  of  program 
evaluation  are  recommended.   The  timing  and  use  of  each  type  is 
determined  from  results  of  previous  evaluations  by  the  ADB  or  MHB 
Bureau  Chief  in  conjunction  with  the  ASD  Administrator  and  the 
program  evaluation  supervisor.   However,  at  least  one  comprehensive 
on-site  visit  per  year  is  recommended. 

-Comprehensive  evaluations  focus  on  all  program  management 
systems.   The  objective  of  these  evaluations  is  to  determine 
whether  the  entire  program  meets  State  and  Federal  standards, 
to  evaluate  the  quality  of  Financial  '''anagement  and  to  vali- 
date reported  client  treatment  data. 

The  time  required  for  a  comprehensive  evaluation  will  depend 
largely  on  program  size  in  terms  of  staff  and  client  volume 
and  the  number  of  program  facilities.   However,  even  in  the 
larger  programs,  comprehensive  evaluations,  including  prepa- 
ration of  the  final  report  should  require  no  more  than  four 
man-weeks  of  effort.   For  example,  a  large  community  mental 
health  center  (excluding  satellites  which  should  be  evaluated 
separately  after  the  center  is  evaluated)  should  require  the 
following  manpov;er: 

.One  person  for  three  weeks  (two  weeks  on-site)  to  conduct 
the  Organization,  Personnel  and  Client  Treatment  systems 
evaluations  including  preparation  of  the  final  report. 

.One  person  for  one  week  on-site  to  conduct  the  Financial 
Management  evaluation  including  preparation  of  the  final 
report . 

Again,  at  least  one  comprehensive  evaluation  of  each  program  should 
be  performed  annually. 

-Interim  evaluations  may  consist  of  either  an  in-depth  evalua- 
tion of  one  or  more  management  systems  or  »    less  intensive 
evaluation  of  all  systems  as  deemed  appropriate  by  ASD  manage- 
ment.  The  objective  of  the  interim  evaluation  is  to  assess 


program  management  progress  during  the  vear  between  compre- 
hensive evaluations.   Three  to  five  man-days  are  generally 
required  to  perform  the  interim  evaluation. 

-Follow-up  evaluations  are  initiated  by  specific  problems 
uncovered  during  a  previous  evaluation.   This  type  will 
usually  consist  of  an  in-depth  review  of  one  or  more  compo- 
nents within  a  particular  management  system,  i.e.  Organiza- 
tion, Personnel,  etc.   The  objective  of  the  follow-up  evalua- 
tion is  to  determine  whether  the  program  has  implemented 
specific  corrective  actions  by  ASD  established  due  dates. 
Normally,  one  day  or  less  is  required  for  a  follow-up  evalua- 
tion. 

It  should  be  noted  that  other  types  of  site  visits  may  be  required 
from  time-to-time  to  provide  programs  with  technical  assistance, 
advice,  or  other  specific  information  such  as  notifying  tlie  program 
(if  new  standards  concerning  program  operations. 

SCHEDULING: 

Each  Bureau  Chief  (ADB  and  MHB)  is  responsible  for  preparing  a 
tentative  schedule  of  evaluations  for  programs  under  his  juris- 
diction.  The  tentative  schedules  should  be  finalized  in  conjunc- 
tion with  the  ASD  Administrator  and  the  program  evaluation  super- 
visor.  Proper  planning  for  comprehensive  evaluations  ensures  that 
sufficient  time  is  available  for  the  site  visit,  that  staff  will  be 
available  and  that  all  programs  are  evaluated  in  a  timely  manner. 
Planning  for  comprehensive  evaluations  occurs  once  a  year.   During 
the  scheduling  phase,  Bureau  Chiefs  should  ensure  that  a  compre- 
hensive evaluation  of  each  program  is  scheduled. 

Interim  evaluations  are  scheduled  on  a  quarterly  basis,  as  needed, 
for  the  subsequent  three  month  period.  The  quarterly  plans  should 
be  consistent  with  annual  plans  to  avoid  duplicate  scheduling. 

Follow-up  evaluations  are  scheduled  to  coincide  with  established 
due  dates  for  corrective  action  resulting  from  comprehensive  or 
Interim  evaluations. 

PRE-PLANNING : 

Pre-planning  is  the  first  phase  of  the  evaluation  process.   This 
phase  includes  three  major  steps:    a)  defining  the  scope  of  the 
evaluation,  b)  making  appointments,  and  c)  reviewing  available 
information  at  the  central  office. 

-Definition  of  Scope: 

During  the  pre-planning  phase,  the  ADB  or  MHB  Bureau  Chief  is 
responsible  for  outlining  to  the  program  evaluation  supervisor 
the  desired  scope  of  work  for  the  on-site  visit.   Scope  depends 
on  the  type  of  evaluation  selected.   For  comprehensive  evalua- 
tions, a  detailed  review  of  all  program  management  systems  is 


required  as  outlined  in  Section  III  of  the  handbook.  For 
interim  or  follow-up  evaluations,  the  areas  to  be  covered 
should  be  specifically  identified. 

-Making  Appointments: 

This  activity  consists  of  determining  the  date  of  the  visit, 
coordinating  the  visit  with  the  designated  program,  making 
appropriate  appointments,  and  confirming  the  evaluation  of 
schedules. 

The  date  and  time  of  the  review  will  depend  upon  the  program's 
normal  working  hours  and  upon  the  availability  of  key  program 
personnel.   In  most  instances,  it  is  advantageous  to  ensure 
the  availability  of  the  program  director  and  other  administra- 
tive personnel  prior  to  the  visit.   Appointments  should  be 
made  with  the  program  director,  either  by  telephone  or  letter, 
several  weeks  in  advance  of  the  visit.   When  making  appoint- 
ments, the  program  evaluator  should  confirm  the  availability 
of  personnel  for  interviews  and  outline  the  scope  and  purpose 
of  the  visit.   This  establishes  a  mutual  understanding  between 
the  director  and  the  evaluator  and  helps  to  ensure  that  on- 
site  review  time  is  most  efficiently  used. 

If  appointments  are  made  by  telephone,  a  letter  should  be  sent 
to  the  program  director  to  ensure  that  both  parties  have 
agreed  on  the  same  date  and  timing.   A  misunderstanding  of  the 
specified  time  and  review  activities  can  present  problems  with 
the  availability  of  key  personnel. 

-Review  Information  in  the  Central  Office: 

Prior  to  the  visit,  the  program  evaluator  should  spend  several 
hours  reviewing  the  program  information  at  the  central  office. 
The  types  of  information  to  be  reviewed  include: 

.Previous  site  visit  reports 
.Pre-audit  and  audit  reports,  if  available 
.Funding  application  or  contracts  and  line  item  budget 
•Program  application  review  comments 
.Voucher  submissions 
. Correspondence 

.The  most  recent  program  report  showing  the  number  of 
clients  served  by  service  program. 

The  extent  of  pre-visit  file  review  will  depend  upon  the 
program  evaluator 's  knowledge  of  the  program,  the  type  of 
review  scheduled,  and  the  elapsed  time  since  the  last  compre- 
hensive review. 

F.     FHTRY  TNTKRVTF;^^ 

The  entry  interview  is  the  second  phase  of  the  program  evaluation. 


Upon  arrival  at  the  program,  the  program  evaluator  will  generally 
contact  the  program  director.   During  the  preliminary  interview 
with  the  director,  the  evaluator  is  responsible  for  defining  his/her 
agenda,  confirming  the  persons  to  be  interviewed,  and  indicating 
the  files  to  be  reviewed.   Specific  topics  to  be  discussed  in  the 
interview  include: 

-Introduction  of  the  evaluation  team  members  and  program  staff. 

-Purpose  of  the  evaluation  and  areas  to  be  covered. 

-An  overview  of  the  program  staff  needed  to  assist  the  evalua- 
tion team. 

-The  date  and  content  of  the  exit  interview. 

-The  timing  and  distribution  of  the  final  report  including  a 
discussion  of  the  debriefing  interview  with  the  ASD  Admini- 
strator . 

-Workspace  for  the  evaluation  team. 

After  the  program  director  and  his  staff  have  had  an  opportunity  to 
question  the  evaluation  team  members  about  the  visit,  the  program 
director  should  be  requested  to  provide  an  overview  of  operations 
covering  the  following  topics: 

-A  description  of  the  organization  chart  including  the  names 
and  locations  of  management  or  supervisory  personnel  needed 
for  the  evaluation  such  as  the  directors  of  training,  per- 
sonnel, client  treatment  and  accounting. 

-An  overview  of  services  provided  and  the  facilities  used. 

-The  location  of  policy  material  such  as  manuals  and  procedures 
to  determine  if  the  review  of  policies  should  be  shared  among 
team  members . 

WORK  PLANNING: 

The  purpose  of  the  work  planning  phase  is  to  determine  the  esti- 
mated time  required  for  evaluation  of  each  management  system  and  to 
assign  specific  duties  to  each  team  member.   Immediately  following 
the  entry  interview,  the  evaluation  team  should  meet  and  discuss 
the  results  of  the  entry  interview.   At  this  meeting,  final  agree- 
ments are  made  about  who  will  conduct  what  portion  of  the  evalua- 
tion and  the  expected  completion  dates.   If  feasible,  an  evaluation 
schedule  should  be  prepared  and  given  to  the  program  director 
showing  expected  times  or  dates  that  various  program  staff  or  facilities 
will  be  contacted  and  by  which  team  members. 

CONDUCTING  THE  EVALUATION: 

During  the  comprehensive  evaluation  of  each  review  area,  the  eval- 
uator should  follow  the  detailed  instructions  and  procedures  out- 
lined in  Section  III  of  the  handbook.   However,  if  additional  methods 
or  procedures  are  necessary,  a  description  of  the  methods  used  should 
be  included  in  supporting  documentation.   Documentation  of  evaluation 
procedures  and  findings  is  viewed  as  an  integral  part  of  the  total 
evaluation  process.   The  effectiveness  of  the  evaluation  effort  is 
directly  related  to  the  quality  and  completeness 


of  evidence  supporting  conclusions  that  a  program  meets  or  does 
not  meet  standards.   In  addition,  the  evaluation  process  results 
m  an  easy  to  prepare  and  accurate  final  report  if^  evaluators 
follow  evaluation  procedures  and  prepare  complete  documentation 
showing  evidence  of  findings. 

Evaluation  procedures  are  organized  to  facilitate  a  step-by-step 
cumulative  understanding  of  a  program's  internal  management  and 
service  delivery  systems.   Following  the  instructions  of  each 
evaluation  guide,  the  evaluator  should  keep  the  following  prin- 
ciples in  mind: 

-Evaluators  should  familiarize  themselves  with  the  content 
of  the  guides  before  use.   Prior  preparation  will  increase  the 
efficiency  in  conducting  procedures  and  the  quality  of  docu- 
mentation. 

-Evaluators  should  follow  the  guides  as  closely  and  se- 
quentially as  feasible  to  help  ensure  that  all  appropriate 
procedures  are  performed  and  questions  asked. 

-In  summarizing  results,  the  evaluators  should  restate  ques- 
tions asked  and  summarize  steps  taken.   For  example,  .  .  . 
"When  asked  who  was  responsible  for  in-service  training,  staff 
indicated  that  the  responsibility  had  not  been  delegated  to 
any  particular  individual."   "The  staff  interviewed  were  .  .  ." 
Procedures  not  performed  or  questions  not  asked  should  also  be 
recorded.   This  method  of  documentation  will  help  ensure  that 
during  final  report  preparation  and  program  debriefing,  the 
evaluator  will  not  have  to  rely  on  memory  of  steps  taken  or 
staff  interviewed  to  interpret  his/her  notes. 

Following  the  above  principles  will  enhance  the  overall  process  of 
conducting  the  evaluation  and  will  result  in  a  well  organized  set 
of  workpapers  for  later  reference  and  completion  of  the  evaluation 
checklist  and  final  report. 

-Use  of  the  Evaluation  Guides  and  Documentation  Worksheets: 

The  four  evaluation  guides  are  intended  for  use  as  a  handbook 
of  instructions  in  performing  a  program  evaluation.   Detailed 
notes  resulting  from  evaluation  procedures  should  be  kept 
separate  from  the  guides  so  that  each  guide  may  be  used  in- 
definitely by  a  program  evaluator.   Several  copies  of  each 
documentation  worksheet,  the  financial  management  internal 
controls  questionnaire  and  the  program  evaluation  checklist 
should  be  contained  in  a  central  file  at  the  ASD  central 
office  for  use  during  an  on-site  evaluation. 

-Organization  of  Workpapers: 

The  workpapers  and  other  documents  supporting  each  separate 
evaluation  should  be  organized  within  five  separate  folders. 


each  labeled  with  the  agency  name  and  dates  and  type  of  re- 
view.  (For  interim  evaluations,  fewer  folders  may  be  re- 
quired, depending  upon  the  scope  of  evaluation.)   The  specific 
content  of  Folder  One  and  general  content  of  Folders  Two 
through  Five  should  be  as  follows: 

.Folder  One,  labeled  "General  Administrative  Summary" 
should  include: 

(-)  A  face  sheet  showing: 

Name  of  Program 
Facility  Evaluated 
Type  of  Evaluation 
Date  of  Evaluation 
Evaluated  By 
Reviewed  By 
Date  of  Last  Evaluation 

(-)  The  Evaluation  Final  Report 

(-)  The  completed  Program  Evaluation  Checklist  for  all 
areas  of  evaluation. 

.Folders  Two  through  Five  should  contain  supporting  docu- 
mentation for  the  findings  using  the  Program  Evaluation 
Guides  and  should,  therefore,  be  labeled  so  that  they 
correspond  to  the  appropriate  guides,  as  follows: 

(-)  Organization  and  Program  Management 

(-)  Personnel  Management  and  Staff  Development 

(-)  Financial  Management 

(-)  Client  Treatment  and  Census 

The  content  of  Folders  Two  through  Five  represent  the 
evaluator's  documentation  of  steps  taken  and  conclusions 
reached  in  the  program  evaluation  process.   As  such,  it 
will  be  used  both  as  the  basis  for  the  preparation  of  the 
Evaluation  Final  Report  and  as  the  justification  to  the 
agency  of  all  statements  made  in  that  report.   It  is, 
therefore,  important  that  documentation  be  precise  and 
well  ordered,  in  a  format  facilitating  final  report 
preparation  and  easy  access  during  agency  debriefing. 

.Each  of  Folders  Two  through  Five  should  contain  all 
supporting  dociomentation  for  the  findings  in  that  area  of 
evaluation.   This  documentation  may  include  interview 
notes,  copies  of  contracts  and  other  official  documents, 
agency  pamphlets  and  manuals,  documentation  worksheets 
completed  per  Guide  instructions,  and  all  other  working 
papers  pertinent  to  the  identified  area  of  evaluation. 

Within  each  of  Folders  Two  through  Five,  papers  should  be 
organized  as  follows: 
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(-)  Sections  should  be  physically  separated  by  14"  x  18' 
two-hole -punched  paper  and  labeled  to  correspond 
with  Guide  and  Checklist  sections.   Each  section 
should  be  given  a  code  letter. 

(-)  Within  sections,  papers  should  be  ordered  sequen- 
tially, i.e.,  in  the  order  of  the  Guide  procedures 
which  they  document,  and  should  be  numbered.   A 
section  code  and  page  nxomber  reference  to  documenta- 
tion can  then  be  recorded  on  the  checklist  for  all 
findings. 

(-)  All  papers  should  be  two-hole-punched  and  fastened 
into  folders. 

(-)  Each  file  should  contain  a  table  of  contents. 

-Program  Evaluation  Checklist: 

A  program  evaluation  checklist  is  provided  in  Appendix  A  of 
the  handbook.   The  checklist  is  designed  to  record  the  results 
of  the  evaluation  to  summarize  recommendations  for  corrective 
action.   Further,  the  checklist  provides  a  column  in  which 
to  record  the  location  of  supporting  evidence  in  evaluation 
workpapers.   The  checklist  is  to  be  completed  after  all  eval- 
uation steps  are  completed  and  is  used  as  a  basis  for  the  exit 
conference  and  final  report.   For  each  area  of  review,  one  of 
three  findings  is  possible: 

.Acceptable  -  No  Action; 

The  program  is  functioning  well  in  this  area  and  there 
are  no  recommendations  for  improvement. 

.Acceptable  -  Action  Recommended: 

The  program's  performance  can  be  improved,  but  currently 
a  deficiency  in  this  area  is  not  interfering  with  program 
operations. 

.Unacceptable  -  Action  Required: 

The  program  is  seriously  deficient  in  an  area  and  cor- 
rective action  is  required. 

All  findings  which  are  unacceptable  should  be  followed 
with  a  summary  statement  of  the  recommended  action.   A 
category  checked  "Acceptable  -  Action  Recommended"  should 
have  a  recommended  action.   There  should  not  be  a  recom- 
mendation after  any  category  checked  "Acceptable  -  No 
Action." 

If  a  category  has  not  been  reviewed,  an  "N/R"  should  be 

shown  in  the  "No  Action"  column.   If  the  category  is  not 

applicable  to  the  type  of  review,  an  "N/A"  should  be 
entered  in  the  "No  Action"  column. 
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EXIT  INTERVIEW: 

The  exit  interview  provides  an  opportunity  for  the  program  eval- 
uator  to  relate  review  findings  to  the  program  director.   An  exit 
interview  is  required  for  each  visit,  although  the  depth  of  the 
interview  will  vary  with  the  type  of  visit. 

In  a  comprehensive  visit,  the  exit  interview  should  begin  with  a 
discussion  of  the  work  performed  as  was  outlined  in  the  entry 
interview.   Then,  the  program  evaluator  should  discuss  the  strengths 
and  weaknesses  found  during  the  review  along  with  specific  recom- 
mendations for  corrective  action.   This  data  should  be  already 
summarized  on  the  evaluation  checklist  as  described  above  and  the 
checklist  should  be  used  for  the  exit  interview.   After  this  dis- 
cussion and  the  resolution  of  any  contended  issues,  the  program 
evaluator  should  again  describe  the  timing  and  distribution  fm  the 
final  report. 

The  interim  evaluation  exit  interview  is  structured  in  the  same  way 
as  ^or  the  comprehensive  evaluation;  however,  strengths  and  weak- 
nesses will  be  limited  to  those  topics  reviewed. 

An  exit  interview  for  a  follow-up  visit  is  used  to  summarize  find- 
ings, and,  if  prior  recommendations  have  not  been  implemented,  to 
report  the  lack  of  program  action  and  the  reasons  given  by  the 
program  director. 

PREPARATION  AND  DISTRIBUTION  OF  THE  FINAL  REPORT:  . 

The  comprehensive  program  evaluation  final  report  is  intended  to  be 
brief  and  outlines  the  evaluation  findings  and  recommendations. 
During  the  field  testing  of  this  handbook,  a  final  report  was  pre- 
pared for  an  alcoholism  treatment  program  and  should  be  used  as  a 
guide  for  all  reports  on  comprehensive  evaluations.   Content  of 
this  report  includes  the  following: 

-Cover  Letter 

-Program  Overview 

A.  General  information  and  services  provided. 

B.  Location  of  facilities  including  program  activities  or 
services  performed  at  each  facility. 

C.  The  present  organization  structure  of  the  program. 

-Findings  and  Recommendations 

A.  Organization  and  Program  Management 

B.  Personnel  Management  and  Staff  Development 

C.  Financial  Management 

D.  Client  Treatment  and  Census 
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-Appendix 

A.  Membership  of  the  Governing  Board 

B.  Estimated  Income  and  Expenses,  1974-1976 

C.  Budgeted  vs.  Actual  Income  and  Expenses,  January-April, 
1976 

D.  Utilization  of  Inpatient  and  Residential  Facilities  and 
Reported  vs.  Documented  Outpatient  Census 

E.  Content  of  Client  Case  Records 

F.  Analysis  of  Support  Service  Contacts 

G.  Analysis  of  Personnel  File  Content 

Findings  of  strengths  and  weaknesses  are  based  on  the  program's 
implementation  of  required  program  standards  as  siommarized  in  the 
evaluation  guides.   Specifically,  if  a  particular  standard  is  being 
met,  it  is  considered  a  program  strength  and  if  not  met,  it  is 
considered  a  weakness.   As  pointed  out  in  Section  II-G  (above) ,  it 
is  extremely  important  that  each  finding  be  supported  by  detailed 
documentation  in  the  program-evaluation  files.   This  is  to  ensure 
that  the  finding  is  based  on  actual  performance  of  evaluation 
procedures  and  can  be  referenced  in  the  event  a  program  director  or 
other  report  readers  desire  further  substantiation  of  the  finding. 

The  final  report  should  be  prepared  in  draft  form  by  the  program 
evaluator  and  submitted  to  the  appropriate  Bureau  Chief  and  ASD 
Administrator  for  review.   Upon  approval  of  report  content  and 
preparation  of  necessary  revisions,  copies  should  be  distributed 
to: 

-The  chairman  of  the  program  governing  or  advisory  board,  and, 
-The  program  director. 

Attached  to  the  report  should  be  a  cover  letter  from  the  ASD  Ad- 
ministrator in  which  a  date  is  established  for  an  evaluation  de- 
briefing to  be  held  at  the  ASD  central  office  (see  below) . 

For  interim  evaluations,  only  those  areas  assessed  should  be  in- 
cluded in  the  report.   Further,  in  place  of  the  "program  overview" 
section,  a  summary  should  be  prepared  outlining  the  major  focus  of 
the  interim  assessment  based  on  the  findings  of  the  previous  com- 
prehensive evaluation. 

For  follow-up  reviews,  a  summary  memorandum  should  be  prepared 
outlining  the  progress  to  date  on  specific  problem  areas  which  were 
to  be  corrected  as  agreed  upon  in  the  evaluation  debriefing. 

CONDUCTING  THE  PROGRAM  EVALUATION  DEBRIEFING: 

The  purpose  of  the  debriefing  interview  is  to  obtain  agreement 
between  ASD  management  and  the  program  director  about  the  prior- 
ities and  due  dates  for  program  corrective  actions.   Attendance 
should  include: 
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-The  ASD  Administrator 

-The  appropriate  Bureau  Chief 

-The  program  evaluation  Unit  Supervisor 

-The  local  Program  Director 

-The  local  program  Board  Chairman  or  Representative 

The  debriefing  occurs  approximately  2-3  weeks  after  the  evaluation 
final  report  has  been  delivered  to  the  program.   The  initial  de- 
briefing activity  is  to  verify  with  the  program  director  that  the 
final  report  accurately  reflects  the  status  of  program  operations 
at  the  time  of  on-site  evaluation.   In  some  cases,  the  program 
director  may  request  further  substantiation  of  report  findings. 
Should  this  occur,  the  program  evaluation  unit  supervisor  should  be 
asked  to  summarize  the  basis  of  findings  as  documented  in  detailed 
notes  in  the  evaluation  files  and  the  source  of  the  indicated 
program  standard. 

After  contended  issues  are  resolved,  the  ADB  or  MHB  Bureau  Chief 
should  outline  which  findings  are  most  critical  requiring  uppermost 
priority  for  corrective  action.   While  all  standards  in  the  hand- 
book are  required  for  program  implementation,  the  following  are 
probably  the  most  urgent  for  correction  to  continue  present  funding 
levels: 

-The  docinnented  client  census  or  active  caseload  should  be  of 
sufficient  size  to  require  present  funding  amounts. 

-Results  of  the  financial  management  evaluation  should  indicate 
that  the  program  can  maintain  required  levels  of  local  cash 
contributions  and  is  adequately  in  control  of  financial  opera- 
tions. 

-The  frequency  of  client  contact  and  quality  of  case  record 
documentation  should  indicate  that  the  program  is  in  control 
of  the  service  delivery  process.   This  is  to  help  insure  that 
clients  are  receiving  necessary  services  and  that  staff  time 
is  efficiently  utilized.   Two  major  aspects  of  this  deter- 
mination is  that  treatment  contacts  are  routinely  documented 
in  case  records  and  that  individualized  client  treatment  plans 
and  progress  notes  are  current. 

-Evaluation  results  of  overall  program  management  systems  such 
as  organization  and  personnel  should  indicate  that  the  program 
Board  of  Directors  and  the  program  director  are  maintaining 
leadership  and  control  of  the  program's  staff  and  operation. 
This  is  indicated  by  sound  personnel  policies  and  staff  per- 
formance evaluation,  continuing  use  of  program  objectives  to 
monitor  program  performance,  and  well  documented  operating 
policies  and  procedures. 

During  the  debriefing  interviev/,  each  weakness  listed  in  the  final 
report  should  be  discussed.   The  program  director  should  explain 
the  planned  corrective  actions  and  expected  due  dates  for  accom- 
plishment.  Agreements  should  be  reached  between  ASD  and  the 
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program  director  about  both  content  and  timing  of  corrective  ac- 
tions.  At  the  conclusion  of  the  debriefing,  the  program  director 
should  be  requested  to  submit  to  the  ADB  or  MHB  Bureau  Chief  a 
follow-up  written  response  to  the  report  outlining  this  corrective 
action  plan.   This  document  should  be  submitted  within  two  weeks 
for  the  debriefing  interview  and  becomes  the  basis  for  subsequent 
interim  and  follow-up  program  evaluations. 
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SECTION  I 


ORGANIZATION  AND  PROGRAM  MANAGEMENT  EVALUATION  GUIDE 


ORGANIZATION  AND  PROGRAM  MANAGEMENT  EVALUATION  GUIDE 

A.    INTRODUCTION: 

Evaluation  of  the  organization  and  program  management  system  in- 
cludes review  of  documents,  interviews  with  staff  and  a  tour  of 
facilities.   The  review  areas  include: 

-organization  structure 
-program  goals  and  objectives 
-policies,  procedures  and  required  plans 
-governing  boards  and  advisory  boards 
-program  self-evaluation 
-facilities 

The  review  should  begin  with  interviews  with  the  program  director 
and  other  appropriate  administrative  personnel  responsible  for 
program  management.   During  the  interviews,  the  evaluator  should 
obtain  a  general  understanding  of  how  the  program  is  managed  by 
obtaining  answers  to  questions  listed  for  each  review  area.   In 
addition,  he  should  identify  the  location  of  supporting  documents 
needed  to  verify  that  requirements  are  being  met. 

B.    ORGANIZATION  STRUCTURE: 

Summary  of  Requirements: 

The  organization  structure  should  be  formally  documented  on  an 
organization  chart  reflecting  current  lines  of  authority  and  re- 
porting relationships.   This  is  to  help  ensure  that  staff  under- 
stand their  position  in  the  organization  and  to  promote  clear 
communications  throughout  the  organization.   To  achieve  this  pur- 
pose, a  current  chart  should  be  available  to  each  staff  member. 

Evaluation  Procedures: 


1 .    Development  and  Communication  of  Organization  Structure: 

a.    Obtain  or  prepare  a  current  organization  chart.   (This 
chart  will  be  included  in  the  final  report.) 
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b.    Interview  the  program  director  and  3-5  staff  and  deter- 
mine the  following:   (See  the  "Professional  Staff  Inter- 
view Questionnaire,"  Exhibit  II-2,  in  the  Personnel 
Management  and  Staff  Development  Evaluation  Guide.) 

.V7hen  was  the  organization  chart  last  updated? 

.Does  the  present  organization  chart  accurately 
reflect  lines  of  authority  and  reporting  relation- 
ships? 

.Are  the  program  director  and  supervisors  available 
to  staff  for  problem  solving  and  direction? 

.Do  staff  believe  the  organization  structure  makes 
sense  and  do  they  believe  that  they  have  been  given 
adequate  instruction  and  authority  to  perform 
assigned  functions? 

.Who  in  the  organization  is  responsible  for  supervi- 
sion of  the  following  functions:   (A  single  indi- 
vidual should  be  assigned  for  each  function.) 

-Personnel 
-Training 
-Accounting 
-Client  Treatment 
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C.    COALS,  OBJECTIVES  AND  PROGRAM  SELF-EVALUATION: 
Summary  of  Requirements: 

A  written  statement  of  goals  and  specific  and  measurable  objectives 
is  to  be  distributed  to  staff  and  made  available  to  clients  and  the 
community.   Monitoring  of  actual  versus  scheduled  progress  in  the 
accomplishment  of  objectives  is  to  be  reported  monthly  to  the 
Department  of  Institutions  in  written  form.   Responsibility  for 
accomplishment  of  objectives  should  be  assigned  to  specific  in- 
dividuals or  organizational  units. 

Governing  boards  are  to  provide  resources  for  the  program  self- 
evaluation  process,  which  is  to  include  at  least  their  annual 
review  of  program  compliance  with  contract  requirements. 

Evaluation  Procedures: 


1 •    Development  of  Goals  and  Objectives: 

To  determine  whether  a  program  has  developed  goals  and  ob- 
jectives, obtain  a  written  statement  of  program  goals  and 
objectives  and  a  schedule  of  dates  for  their  accomplishment. 
Interview  the  program  director  and  determine  the  following: 

.How  are  program  goals  and  objectives  established? 

.Do  they  reflect  efforts  to  improve  or  enhance  past  per- 
formance? 

2 .    Communication  of  Coals  and  Objectives: 

To  assess  communication  of  goals  and  objectives  to  staff, 
clients  and  the  community  interview  the  program  director  and 


1-3 


3-5  professional  staff  to  obtain  the  following  information: 
(See  the  "Professional  Staff  Interview  Questionnaire,"  Exhibit 
II-2,  in  the  Personnel  Management  and  Staff  Development 
Evaluation  Guide.) 

a.  Are  staff  aware  of  current  goals  and  objectives?   Do  they 
believe  the  statement  accurately  reflects  current  goals 
and  objectives  based  on  directions  received? 

b.  What  means  are  used  to  communicate  goals  and  objectives 

to  clients  and  the  community?   Review  printed  materials  such 
as  pamphlets  or  brochures  available  to  clients  and  the 
community. 

3 .    Monitoring  of  Goals  and  Objectives: 

To  determine  whether  formal  monitoring  o^   goals  and  objectives 
is  done,  interview  the  program  director  and  determine  the 
following: 

a.  To  whom  are  responsibilities  for  achieving  and  monitoring 
objectives  assigned? 

b.  Review  files  of  progress  reports  to  confirm  that  objec- 
tives are  formally  monitored  for  achievement  at  least 
monthly . 
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c.    Review  documentation  of  follow-up  action  taken  on  find- 
ings reported  in  progress  reports.   For  objectives  not 
achieved  by  expected  due  dates,  are  new  due  dates  estab- 
lished? 

4.    Performance  of  Program  Self-Evaluation: 

To  determine  whether  the  program  conducts  required  formal 
self-evaluation  of  its  compliance  with  contract  requirements, 
interview  the  program  director  and  determine  the  following: 

a.  What  procedures  are  used  for  self -evaluation? 

b.  Are  formal  reports  prepared  and  submitted  to  the  govern- 
ing or  advisory  board? 

c.  Review  written  reports  of  the  most  recent  self-evaluation. 
Is  self-evaluation  performed  at  least  annually? 


D.    POLICIES,  PROCEDURES  AND  REQUIRED  PLANS: 

Summary  of  Requirements: 

Written  statements  of  program  policies,  procedures  and  plans  are  to 
be  available  or  distributed  to  staff  as  appropriate  in  a  format 
amenable  to  formal  and  regular  revision  and  maintenance.   The 
statements  are  to  include,  for  all  programs: 

-Plan  for  Affirmative  Action  (if  the  facility  receives  direct 
federal  funds)  including  recruiting  and  hiring  policies. 

-Annual  plan  for  organized  in-service  training,  including  on- 
going training,  orientation  for  new  employees,  and  supervision 
of  staff;  and  a  policy  on  educational  leave. 
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-Standards  (or  qualification  requirements)  for  professional  and 
trained  non-professional  staff. 

-Policy  of  non-discrimination  among  clients. 

-Accounting  policies  and  procedures  as  required  by  the  Depart- 
ment of  Institutions. 

-Individualized  Treatment  Plan  for  each  client. 

-Program  service  procedures  for: 

Admissions 

Screening 

Referral  to  other  agencies  and  current  resource  listings 

Aftercare  and  follow-up 

Emergency  treatment 

-Policies  for  personnel  which  should  include  procedures  for 
recruitment,  selection  and  termination  of  staff  and  for: 

.Wage  and  Salary  Administration 

.Employee  Benefits 

.VJorking  Hours 

.Vacation  and  Sick  Leave 

.Rules  of  Conduct  and  Disciplinary  Action 

.Work  Performance  Evaluation 

.Arbitration  of  Employee  Grievances 

Additional  Requirements  for  Drug  Abuse  Treatment  Programs  are: 

-Policy  for  client  termination  including  termination  criteria 
and  recording  requirements. 

-^''ormal  plan  for  drug  abuse  prevention  and  education.   The  plan 
is  to  include: 

.Defined  Target  Groups 

.Goals  and  Objectives  of  the  Program 

.Designated  Personnel  for  Program  Implementation 

.Program  Content 

.Methods  to  be  Used 

.Evaluation  Procedures 

Additional  Requirements  for  Alcoholism  Treatment  Programs  arc: 

-Policies  for  overall  operation  including  scope  of  services, 
groups  to  be  served,  and  methods  of  service. 

-Policy  for  client  transportation. 

-Policy  for  client  termination  including  termination  criteria 
and  recording  requirements. 
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Residential  alcoholism  treatment  programs  are  required  to 
have  : 

.Schedule  for  recreation  and  rehabilitation  programs. 

.Policy  on  "length  of  stay". 

.Policies  and  procedures  for  the  distribution  of  medi- 
cation, administration  of  first  aid  and  referral  for 
treatment  of  medical  emergencies. 

Evaluation  Procedures: 


1-    Development  of  Required  Program  Policies,  Procedures  and  Pi 


ans : 


To  assess  the  program's  policy  making  process,  and  to  deter- 
mine whether  the  program  has  developed  required  written  state- 
ments of  policy,  procedures  and  plans,  interview  the  program 
director  to  deteimine  the  following: 

a.  Who  is  responsible  for  establishing,  changing  and  ap- 
proving program  policies? 

b.  What  is  the  role  of  the  governing  board  in  the  policy 
making  process?   During  the  review  of  board  meeting 
minutes  (see  below)  verify  that  the  board  actively 
participates  in  policy  development  and  approval. 

c.  Are  there  written  statements  of  program  policies,  pro- 
cedures and  plans?   Obtain  and  reviev;  copies  of  those 
required  and  document  their  existence  on  the  "Policies 
Documentation  Worksheet,"  Exhibit  I-l,  indicating  the 
titles  of  documents  and  the  date  they  were  prepared. 
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NOTE:      If  copies  of  the  required  written  policies, 

procedures  and  plans  are  not  centrally  located, 
the  evaluator  should  consider  requesting  that 
another  member  of  the  evaluation  team  verify 
their  existence.   For  example,  existence  of 
personnel  policies  might  be  verified  by  the 
team  member  performing  the  evaluation  of  per- 
sonnel management  and  staff  development. 
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EXHIBIT  I-l 
POLICIES  DOCUMENTATION  WORKSHEET 


PROGRAM  NAME: 
REVIEWED  BY: 
DATE: 


REQUIRED  POLICY/PROCEDURE/PLAN         DOCUMENT  TITLE         DATE  PREPARED 
All  Programs 

-Plan  for  Affirmative  Action 
(if  the  facility  receives 
direct  federal  funds)  includ- 
ing recruiting  and  hiring 
policies . 

-Annual  plan  for  organized  in- 
service  training,  including 
on-going  training,  orienta- 
tion for  new  employees,  and 
supervision  of  staff;  and  a 
policy  on  educational  leave. 

-Standards  for  professional  and 
trained  non-professional  staff. 

-Policy  of  non-discrimination 
among  clients. 

-Accounting  policies  and  pro- 
cedures as  required  by  the 
Department  of  Institutions. 

-Individualized  Treatment  Plan 
for  each  client. 

-Program  service  procedures  for: 

Admissions 

Screening 

Referral  to  other  agencies 

and  current  resource 

listings 
Aftercare  and  follow-up 
Emergency  treatment 

-Policies  for  personnel  which 
should  include  procedures  for 
recruitment,  selection  and 
termination  of  staff  and 
for: 
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REQUIRED  POLICY/PROCEDURE/PLAN         DOCUMENT  TITLE         DATE  PREPARED 

.Wage  and  Salary  Admini- 
stration 

.Employee  Benefits 

.Working  Hours 

.Vacation  and  Sick  Leave 

.Rules  of  Conduct  and 
Disciplinary  Action 

.Work  Performance  Evalua- 
tion 

.Arbitration  of  Employee 
Grievances 

Additional  Requirements  for  Drug 
Abuse  Treatment  Programs  are: 

-Policy  for  client  termina- 
tion including  termination 
criteria  and  recording  re- 
quirements. 

-Formal  plan  for  drug  abuse 
prevention  and  education. 
The  plan  is  to  include: 

.Defined  Target  Groups 
.Goals  and  Objectives  of 

the  Program 
.Designated  Personnel  for 

Program  Implementation 
.Program  Content 
.Methods  to  be  Used 
.Evaluation  Procedures 

Additional  Requirements  for 
Alcoholism  Treatment  Programs 
are: 

-Policies  for  overall  client 
treatment  including  scope  of 
services,  groups  to  be  served, 
and  methods  of  service. 

-Policy  for  client  transporta- 
tion. 

-Policy  for  client  termina- 
tion including  termination 
criteria  and  recording 
requirements . 
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REQUIRED  POLICY/PROCEDURE/PLAN         DOCUMENT  TITLE         DATE  PREPARED 

-Residential  alcoholism 
treatment  programs  are  re- 
quired to  have: 

.Schedule  for  recreation 
and  rehabilitation  pro- 
grams . 

•Policy  on  "length  of 
stay". 

.Policies  and  procedures  for 
the  distribution  of  medica- 
tion, administration  of 
first  aid  and  referral  for 
treatment  of  medical  emer- 
gencies. 
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2.   Communication  of  Program  Policies  and  Procedures: 

To  assess  the  extent  to  which  program  policies  and  procedures 
are  communicated  to  staff,  interview  the  program  director  and 
determine  the  following: 

a.  Describe  how  new  and  revised  policies  are  communicated  to 
staff? 

b.  Does  a  policy  manual  exist  covering  the  major  aspects  of 
the  program?   Summarize  its  contents,  i.e.,  list  the 
general  areas  it  covers. 

c.  To  whom  is  the  manual  distributed? 

d.  Is  the  manual  format  amenable  to  formal  and  regular 
revision  and  maintenance?   Assess  whether  the  manual 
provides  an  overall  structure  within  which  individual 
policy  or  procedural  changes  can  be  documented.   The 
manual  should  not  be  just  a  looseleaf  series  of  policy 
memos . 

e.  Are  policies  and  procedures  clearly  communicated  to 
staff?   Interview  one  staff  member  from  each  of  the  fol- 
lowing staff  classifications:    clerical,  administrative, 
treatment,  to  assess  this  communication.   (See  the  "Pro- 
fessional Staff  Interview  Ouestionnaire,"  Exhibit  II-2, 
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in  the  Personnel  Management  and  Staff  Development  Evalu- 
ation Guide.) 


1^.    GOVKRNING  BOARDS  AND  ADVISORY  BOARDS: 

Summary  of  Requirements: 

Each  non-government  agency  is  to  have  a  governing  board  with  the 
responsibility  to : 

-Develop  and/or  approve  by  laws  and  policies  and  annual  agency 
plans. 

-Receive  and  administer  agency  funds  and  resources. 

-Meet  quarterly  (monthly  for  mental  health  agencies)  and  keep 
meeting  minutes. 

-Employ,  supervise  and  evaluate  the  agency  director. 

-Be  representative  of  the  community  in  membership. 

Each  community-based,  government-administered  drug  abuse  treatment 
or  mental  health  agency  is  to  have  an  advisory  board,  with  the  re- 
sponsibility to: 

-Assist  the  agency  director  or  administrator  in  development  of 
program  policy. 

-Review  annual  agency  plans  and  budget. 

-Meet  quarterly  and  keep  meeting  minutes. 

-Be  representative  of  the  community  in  membership. 

Evaluation  Procedures : 

1-    Presence  and  Role  of  Governing  Board: 

To  determine  whether  a  non-government  agency  has  a  governing 
board,  and  to  assess  the  role  it  plays  in  directing  and  moni- 
toring the  agency,  perform  the  following  steps: 
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a.  Interview  the  agency  director  to  determine  whether  the 
agency  has  a  governing  board,  and  how  often  they  meet. 
Obtain  a  list  of  board  membership  and  each  members  title 
or  background,  and  identify  the  location  of  minutes  of 
board  meetings. 

b.  Interview  the  chairman  of  the  governing  board  to  verify 
the  role  of  the  board  in  monitoring  and  directing  the 
agency,  e.g.,  fiscal  accountability,  policy  development 
or  approval ,  etc . 

2 .    Frequency  and  Documentation  of  Board  Meetings : 

a.  Review  minutes  of  the  governing  board  or  advisory  board 
meetings  for  the  last  six  months  to  confirm: 

.Frequency  of  meetings 

.Attendance 

.Role  of  the  board  in  program  management.   Document 
examples  of  board  actions  v?hich  indicate  that  the 
board  participate  in  major  agency  decisions  such  as 
budget  preparation,  policy  development  and  approval, 
and  approval  of  unbudgeted  expenditures. 

b.  For  each  meeting,  note  whether  at  least  two- thirds  of 
board  membership  was  in  attendance. 
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3-    Presence  and  Role  of  Advisory  Board  and  Documentation  of 
Advisory  Board  Meetings: 

To  determine  that  a  community-based,  government-administered 
agency  has  an  advisory  board  and  to  assess  what  role  it  plays 
in  guiding  and  advising  the  program,  conduct  the  same  steps  as 
in  IE  1  and  2. 

FACILITIES: 

Summary  of  Requirements: 

Program  facilities  are  to  be  licensed  under  Montana  Administrative 
Code,  Chapter  22,  Section  16-2.22(1)-S2210,  by  the  Department  of 
Health  and  Environmental  Sciences.   Facilities  are  to  be  well- 
maintained  and  free  from  obvious  hazards  such  as  broken  windows  and 
stair  cases.   Facilities  should  provide  adequate  space  for  staff 
and  client  treatment  activities.   Alcohol  programs  are  to  have 
liability  insurance  of  at  least  $300,000  on  their  total  operations, 
including  client  transportation. 

Evaluation  Procedures: 
1 .    licensure: 

To  determine  the  program's  compliance  with  licensure  require- 
ments, observe  whether  the  program  has  a  current  DHES  license. 
If  not,  review  the  DHES  inspection  documents  and  note  the 
reasons  that  a  license  was  not  granted. 

2 .    Adequacy  of  Facilities: 

To  determine  whether  facilities  provide  adequate  space,  are 
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well-maintained  and  generally  safe,  perform  the  following 
steps : 

.Conduct  a  tour  of  the  treatment  facility  to  observe 
general  maintenance,  utilization  of  space,  and  security 
of  client  files,  equipment  and  drugs. 

.Interview  3-5  professional  staff  to  determine  whether,  in 
their  judgment,  space  for  client  reception  and  interview- 
ing is  adequate.   (See  the  "Professional  Staff  Interview 
Questionnaire,"  Exhibit  II-2,  in  the  Personnel  Management 
and  Staff  Development  Evaluation  Guide.) 

3.    Insurance  Coverage: 

To  determine  whether  alcohol  programs  are  insured  as  required, 
observe  the  existence  of  a  liability  insurance  policy  of  at 
least  $300,000  on  the  total  operation,  including  client  trans- 
portation. 
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SECTION  II 


PERSONNEL  MANAGEMENT  AND  STAFF  DEVELOPMENT  EVALUATION  GUIDE 


II. 

PERSONNEL   MANAGEfffiNT   AND   STAFF    DEVELOPMENT   EVALUATION   GUIDE 


INTRODUCTION: 

Evaluation  of  the  personnel  and  staff  development  system  includes 
interviews  with  staff,  review  of  personnel  files  and  review  of 
inter-agency  agreements  and  contracts.   The  evaluation  areas  are: 

-job  descriptions,  staff  certification 

-personnel  files  and  staff  performance  evaluation 

-staff  development 

-use  of  volunteers 

-turnover,  vacancies  and  staff  availability 

-subcontracts  and  service  agreements 

The  review  should  begin  with  an  interview  with  the  staff  member (s) 
responsible  for  personnel  policies,  staff  development  and  develop- 
ment of  inter-agency  service  agreements.   During  initial  inter- 
views, the  following  areas  should  be  included: 

-Determine  the  approximate  number  of  professional  staff  "certi- 
fied" by  the  Department  of  Institutions. 

-Determine  how  personnel  files  are  organized  including  location 
and  content. 

-Determine  the  nature  of  the  program's  staff  development  plans 
and  activities. 

-Determine  whether  volunteers  are  being  used. 

-Determine  the  number  of  agencies  with  which  the  program  has 
developed  agreements  or  contracts  for  provision  of  services  to 
program  clients. 

During  this  interview,  the  evaluator  should  explain  the  purpose  and 
methods  of  the  personnel  and  staff  development  review.   Following 
the  interview,  the  evaluator  should  perform  the  following  evalua- 
tion procedures. 

NOTE:     If  copies  of  written  program  policies,  procedures 
and  plans  are  not  centrally  located,  additional 
evaluation  procedures  may  be  required.   (See  the 
Organization  and  Program  Management  Review  Guide . ) 
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JOB  DESCRIPTIONS  AND  STAFF  CERTIFICATION: 

Summary  of  Requirements : 

For  all  mental  health,  drug  abuse  treatment  and  alcoholism  treat- 
ment staff  positions : 

.Job  descriptions,  including  required  staff  qualifications  must 
be  prepared. 

•A  position  classification  system  must  be  developed  which  dif- 
ferentiates between  levels  of  responsibility  and  complexity  of 
work. 

.For  professional  positions,  personnel  must  meet  the  Department 
of  Institutions  certification  standards  and  be  appropriately 
certified . 

Evaluation  Procedures: 

1.    Development  of  Written  Job  Descriptions  and  Staff  Qualifications: 


To  assess  whether  the  program  systematically  develops  required 
qualifications  for  and  descriptions  of  each  staff  position, 
interview  the  personnel  director  and  perform  the  following 
steps : 

a.  Describe  the  procedures  used  to  hire  new  personnel  in- 
cluding : 

. Application  for  employment 

.Persons  responsible  for  selecting  new  employees 

.Position  classification  and  qualifications 
.Job  descriptions 

b.  Briefly  review  resumes  of  newly  hired  personnel  to  assess 
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whether  education  and/or  experience  meets  qualification 
standards. 

c.    From  the  organization  chart,  select  every  third  position 
and  observe  a  written  copy  of  the  job  description  for 
that  position.   Document  results.   Keep  a  record  of  these 
positions  for  further  testing  in  the  following  evaluation 
procedures. 

2 .    Professional  Staff  Certification: 

To  determine  whether  professional  staff  are  certified  as 
required,  observe  the  certification  document  issued  by  the 
Department  of  Institutions  for  all  professional  staff  selected 
in  the  above  sample  of  staff  positions.   Document  the  number 
and  percent  of  those  positions  for  which  staff  are  certified. 
For  those  not  certified,  has  an  application  for  certification 
been  sent  to  the  department? 

PERSONNEL  FILES  AND  STAFF  PERFORMANCE  EVALUATION: 
Summary  of  Requirements: 

For  all  mental  health,  drug  abuse  treatment  and  alcoholism  treat- 
ment programs,  a  personnel  file  is  to  be  maintained  for  each  em- 
ployee.  Further,  a  system  should  be  implemented  for  staff  per- 
formance evaluation  at  least  annually.   While  not  required,  content 
of  personnel  files  should  include  at  a  minimum: 

.Employee  resume  and  job  application 

.Compensation  amount  including  historical  documentation  for 
increases 
.Results  of  performance  evaluation 
.W-4  form 

.For  terminated  employees,  reasons  for  termination. 
.Job  description 

II-3 


Evaluation  Procedures: 

1 ,    Implementation  of  Personnel  Files  and  Staff  Performance 
Evaluation : 

To  determine  the  existence  and  content  of  personnel  files  and 
whether  staff  performance  evaluations  are  routinely  conducted, 
interview  the  personnel  director  and  obtain  access  to  person- 
nel files  and  perform  the  following  steps: 

a.  Describe  the  program's  procedure  for  conducting  staff 
performance  evaluations  including: 

.Frequency 

.Content  and  forms  used  (obtain  a  copy) 

.Personnel  responsible  for  conducting  staff  per- 
formance evaluation 

.Is  the  agency  director's  performance  annually 
evaluated  by  the  governing  board? 

NOTE:     Not  applicable  to  government  agencies. 

b.  For  the  employees  selected  in  the  above  sample  of  staff 
positions,  obtain  and  review  each  personnel  file  (include 
the  program  director)  and  complete  the  "Personnel  File 
Documentation  Worksheet"  Exhibit  II-l.   Also,  obtain  the 
personnel  file(s)  for  recently  terminated  employees  and 
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b.  Obtain  files  indicating  training  content  and  attendance 
of  training  programs  and  document  the  following: 

.Approximately  what  percent  of  the  then  current 
professional  staff  attended  each  training  program? 

.Were  written  outlines  and  training  materials  de- 
veloped indicating  a  well  planned  training  activity? 

c.  Describe  the  system  for  training  new  employees.   Has  the 
program  developed  staff  orientation  materials  for  all  new 
employees  such  as  a  staff  handbook? 

d.  To  validate  the  existence  of  orientation  and  on-going 
training,  interview  3-5  staff  members  and  determine  the 
following:   (See  the  "Professional  Staff  Interview  Ques- 
tionnaire," Exhibit  II-2.) 

.Did  all  staff  interviewed  participate  in  orientation 
training? 

.Did  professional  staff  have  adequate  notice  and 
opportunity  to  attend  on-going  training?   Did  pro- 
fessional staff  attend  training  programs  provided 
during  the  last  12  months? 

.For  mental  health  programs,  did  professional  staff 
have  an  opportunity  to  attend  three  in-service 
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training  seminars  or  workshops  and  one  external 
training  program  during  the  last  12  months? 


E  .    USE  OF  VOLUNTEERS: 

Summary  of  Requirements: 

For  all  mental  health,  drug  abuse  treatment  and  alcoholism  treat- 
ment programs,  volunteer  selection  and  evaluation  criteria  should 
be  written  and  distributed  to  volunteers.   Volunteers  should  be 
trained  for  their  assignments  and  supervised  by  full-time  profes- 
sional staff.   And  finally,  (while  not  a  requirement),  records 
should  be  kept  of  hours  spent  by  each  volunteer. 

NOTE:  Record  keeping  of  hours  contributed  by  volunteers  is^ 
a  requirement  for  those  programs  using  volunteers  as 
in-kind  contributions  for  matching  purposes. 

Evaluation  Procedures: 


1 .    Use  of  Volunteers: 

To  assess  the  extent  to  which  volunteers  are  used  and  the 
adequacy  of  their  training  and  supervision  interview  the 
Individual  responsible  for  volunteer  selection  and  monitoring 
and  perform  the  following  steps: 

a.  Describe  the  services  rendered  by  volunteers. 

b.  Obtain  and  review  documentation  describing  volunteer 
selection  and  evaluation  criteria  for  each  service  which 
is  provided  by  volunteer  staff. 

c.  Describe  the  volunteer  training  procedures  used  including 
description  of  printed  materials  distributed  to  volunteers, 
Does  it  appear  that  volunteer  training  is  well  planned? 
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d.    Interview  3-5  volunteers  and  determine  the  following: 

-In  the  volunteer's  opinion,  did  each  volunteer  re- 
ceive adequate  training  for  his/her  assignments? 

-Does  the  volunteer  have  adequate  resources,  supervi- 
sion and  easy  access  to  professional  staff  for  con- 
sultation? 

-What  methods  are  used  for  recording  of  daily  hours 
contributed  by  the  volunteer? 

2.    Record  Keeping  for  Staff  Hours  Contributed  by  Volunteers: 

To  assess  the  adequacy  of  record  keeping  for  contributed  staff 
hours  by  volunteers  especially  for  those  programs  using  vol- 
unteers as  "in-kind  match"  for  state  or  federal  matching 
requirements,  interview  the  appropriate  administrative  staff 
and  determine  the  following: 

a.  Are  volunteers  used  for  in-kind  matching  purposes? 

b.  Describe  the  system  for  reporting  and  recording  of  con- 
tributed volunteer  hours.   Is  the  system  easy  to  follow 
and  is  there  appropriate  supporting  documentation  for 
hours  spent  during  the  most  recent  month?   For  three 
volunteers,  trace  the  reported  hours  spent  during  the 
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most  recent  month  back  to  time  cards  or  logs  maintained 
by  volunteers. 

F.    TURN  OVER  RATE,  VACANCIES  AND  STAFF  AVAILABILITY: 

Summary  of  Requirements:  , 

For  all  mental  health,  drug  abuse  treatment  and  alcoholism  treat- 
ment programs,  professional  staff  time  should  be  available  to  the 
extent  of  caseload  demand.   This  means  that  client-requests  for 
services  should  be  met  on  a  timely  basis  indicated  by  minimum  needs 
for  "waiting  lists";  and  professional  staff  ability  to  provide 
services  requested  by  their  current  caseload.   Other  indicators  of 
possible  lack  of  staff  availability  are  6  month  turn  over  rates  of 
more  than  20-30%  and  excessive  delays  (more  than  two  months)  in 
filling  vacant  professional  staff  positions. 

Evaluation  Procedures: 


1 .   Waiting  Lists  and  Professional  Staff  Availability: 

For  this  procedure  and  all  following  related  procedures,  the 
evaluator  should  attempt  to  gain  an  overall  impression  of 
whether  or  not  there  is  a  major  program  problem  with  staff 
availability  to  clients.   These  procedures  are  not  intended  to 
obtain  "absolute"  determination  of  the  lack  of  staff  avail- 
ability due  to  the  subjective  nature  of  the  evaluation  area. 
Rather,  by  performing  the  following  steps,  insights  should  be 
available  for  discussion  with  the  program  director  about 
possible  corrective  actions  to  improve  overall  staff  avail- 
ability if  necessary.   In  order  to  gain  these  insights,  per- 
form the  following  steps. 


11-10 


a.  Interview  the  program  receptionist  or  other  appropriate 
clerical  personnel  and  determine  the  following: 

.Are  waiting  lists  used?   If  yes,  how  many  clients 
are  on  the  list  waiting  for  assignment  to  a  pro- 
fessional for  intake  or  counseling  activities? 

.In  the  receptionist's  opinion,  is  it  routinely  dif- 
ficult to  find  a  professional  available  for  assign- 
ment of  new  clients  or  one-time  requests  for  ser- 
vices? 

b.  Interview  3-5  professional  staff  representing  each  ser- 
vice program  available  (residential,  outpatient,  partial 
hospitalization,  etc.)  and  determine  the  following:   (See 
the  "Professional  Staff  Interview  Questionnaire,"  Exhibit 
II-2.) 

.Do  staff  believe  they  are  generally  available  to 
meet  client-requests  for  services? 

.If  no,  determine  the  reason  and  document  results. 
Compare  findings  with  documented  client  census  data. 
(See  the  client  treatment  and  census  review  guide.) 

NOTE:     The  evaluator  should  be  cautious  in 
assessing  the  reasons  given  and 
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determine  if  the  problem  relates  to  a 
general  excessive  caseload  or  poor 
counselor  assignment  practices.) 

2.    Turn  Over  Rate: 

To  determine  the  approximate  turn  over  rate  for  professional 
treatment  staff  (as  opposed  to  clerical  or  purely  admini- 
strative) obtain  from  the  personnel  director  the  present 
number  of  professional  treatment  staff  and  the  number  term- 
inated during  the  past  six  months.   Compute  the  six  month  turn 
over  rate  as  follows: 

Number  of  professional  staff  termi- 
nations in  the  last  six  months        X   100  =  7o 


Number  of  current  professional  staff 
Vacancies : 

To  assess  the  extent  to  which  program  management  aggressively 
pursues  filling  vacant  professional  positions,  interview  the 
personnel  director  and  review  the  program  personnel  budget  and 
personnel  or  payroll  records  to  determine  the  following: 

a.    How  many  budgeted  professional  treatment  positions  are 

currently  vacant?   List  each  position  including  budgeted 
salary  and  required  education  or  experience. 
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b.  For  each  vacant  position,  determine  the  length  of  time 
the  position  has  been  vacant. 

c.  For  those  vacant  over  two  months,  determine  the  reason 
and  describe  the  personnel  director's  procedures  for 
filling  vacant  positions.   Does  the  problem  appear  to  be 
related  to  the  lack  of  available  qualified  personnel  or 
to  general  weaknesses  in  the  timeliness  or  regularity  of 
pursuing  new  employees? 

4 .    General  Assessment: 

At  the  conclusion  of  all  the  above  staff  availability  evalua- 
tion procedures,  the  evaluator  should  make  a  general  assess- 
ment about  the  reasonableness  of  present  staffing  levels 
considering  the  following  areas: 

.Does  the  program  appear  to  be  over  budgeted  for  personnel 
costs  considering  the  actual  number  of  filled  versus 
vacant  positions? 

.Does  the  documented  caseload  support  the  need  for  current 
budgeted  staff  positions?   (See  Client  Treatment  and 
Census  Review  Guide.) 
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EXHIBIT  II-2 


P ROFESSIONAL  STAFF  INTERVIEW  QUESTIONNAIRE 


The  questions  below  are  to  be  asked  of  3  to  5  staff  members  represent- 
ing all  service  programs  available  (residential,  outpatient,  partial 
hospitalization,  etc.).   The  purpose  of  the  questionnaire  is  to  verify 
with  staff  that  program  requirements  for  selected  areas  have  been 
implemented.   Prior  to  conducting  the  staff  interview,  the  evaluator 
should  meet  with  other  members  of  the  evaluation  team  to  obtain  the 
necessary  background  information  indicated  by  each  question. 


I.  ORGANIZATION  AND  PROGRAM  MANAGEMENT 

A.    Organization  Structure 

1.    Does  the  present  organization  chart 
accurately  reflect  lines  of  author- 
ity and  reporting  relationships? 

YES         NO 

2.   Are  the  program  director  and  super- 
visors available  to  staff  for  prob- 
lem solving  and  direction? 

YES         NO 

3.    Do  staff  believe  the  organization 
structure  makes  sense  and  do  they 
believe  that  they  have  been  given 
adequate  instruction  and  authority 
to  perform  assigned  functions? 

YES         NO 

4.   Who  in  the  organization  is  respon- 
sible for  supervision  of  the  fol- 
lowing functions:    (A  single 
individual  should  be  assigned  for 
each  function.) 

— Personnel 

— Training 

— Accounting 

— Client  Treatment 
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B.    Goals,  Objectives,  and  Program  Self- 
Evaluation 

1  .    Are  staff  aware  of  current  program 
goals  and  objectives?   Do  they  be- 
lieve, based  upon  directions  they 
have  received,  that  the  written 
statement  of  goals  and  objectives 
accurately  reflects  current  goals 
and  objectives? 

YES         NO 

C.    Facilities 

1.    In  the  judgment  of  staff,  is  space 
for  client  reception  and  interview- 
ing adequate? 

YES         NO 

TI.  PERSONNEL  MANAGEMENT  AND  STAFF  DEVELOPMENT 

A.    Staff  Development 

1.    Did  all  staff  interviewed  partici- 
pate in  orientation  training? 

YES         NO 

2.    Did  professional  staff  have  ade- 
quate notice  and  opportunity  to 
attend  ongoing  training?   Did 
professional  staff  attend  train- 
ing programs  provided  during  the 
last  twelve  months? 

YES         NO 

3.    (For  mental  health  programs)  Did 
professional  staff  have  an  oppor- 
tunity to  attend  three  in-service 
training  seminars  or  workshops 
and  one  external  training  program 
during  the  last  twelve  months? 

YES         NO 

B.    Turnover  Rate,  Vacancies  and  Staff 
Availability 

1.    Do  staff  believe  they  are  generally 
available  to  meet  client  requests 
for  service? 

YES         NO 
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III.  CLIENT  TREATMENT  AND  CENSUS 

A.    Client  Census  and  Reporting 


1.    Are  program  reporting  and  recording 
procedures  understood  by  staff? 


YES 


NO 


NOTE: 


Explain  All  NO  Answers 
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G.    SUB-CONTRACTS  AND  SERVICE  AGREEMENTS: 

Summary  of  Requirements: 

For  mental  health,  drug  abuse  treatment  and  alcoholism  treatment 
programs,  agreements  or  contracts  with  other  service  agencies  pro- 
viding services  for  which  the  agency  pays  should,  at  a  minimum, 
include: 

.Description  of  services 

.Basis  for  payment  including  number  of  clients  to  be  served  and 
total  amount  of  the  contract 

.Duration  of  the  contract 

.Appropriate  signatures  of  the  program  director  and  a  repre- 
sentative of  the  board  of  directors 

Evaluation  Procedures: 


1 .    Existence  and  Content  of  Service  Agreements  and  Contracts: 

To  assess  the  existence  and  nature  of  service  agreements, 
interview  the  program  director  and  obtain  copies  of  contracts 
or  agreements  for  services  for  which  the  agency  pays.   Deter- 
mine the  following: 

a.  What  services  are  provided  under  agreement  or  contract 
and  by  what  agencies?   At  what  cost? 

b.  Are  written  contracts  or  service  agreements  developed  for 
each?   Do  they  contain  the  suggested  content?   Document 
results. 

c .  Does  the  agency  document  services  actually  received  under 
contract?   Are  there  attachments  to  invoices  which  indi- 
cate what  services  were  provided? 
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SECTION  III 


FINANCIAL  MANAGEMENT  EVALUATION  GUIDE 


Ill, 


FINANCIAL  MANAGEMENT  EVALUATION  GUIDE 


INTRODUCTION: 

Review  of  the  financial  management  practices  of  mental  health,  drug 
abuse  treatment  and  alcohol  treatment  programs  includes  interviews 
with  the  program  director  and  the  chief  fiscal  officer,  documenta- 
tion of  total  expenditures  and  revenue  for  the  present  program  year 
and  the  past  three  program  years,  and  assessment  of  internal  finan- 
cial controls  used  by  the  program.   This  assessment  will  involve 
detailed  review  of  the  programs  internal  accounting  system  and  as  a 
result  requires  the  skills  of  a  trained  and  experienced  individual 
in  financial  accounting.   The  review  areas  include: 

-Current,  historical  and  future  financial  condition. 

-Budgeting  performance  and  cash  management. 

-Review  of  internal  controls  including: 

.General  accounting  issues 

.Payroll  and  personnel 

.Procurement  and  payments 

.Supplies  inventory 

.Revenue  sources,  billing  and  collections 

.Petty  cash  and  cash  on  hand 

.Pre-paid  expenses  and  other  current  assets 

.Property  plant  and  procurement 

.Notes  payable  and  long  term  debt 

If  the  fiscal  system  is  maintained  by  an  agency  of  county  or  state 
government,  the  review  of  internal  controls  will  not  be  conducted. 

The  review  should  begin  with  an  interview  of  the  program's  chief 
fiscal  officer.   The  objectives  of  the  initial  interview  are  to: 

-Identify  other  fiscal  personnel  to  be  interviewed. 

-Identify  sources  of  program  funds  by  type  and  funding  period. 

-Determine  location  and  content  of  a  fiscal  policy  manual  or 
procedures . 

-Determine  whether  the  program  is  receiving  timely  reimburse- 
ment from  federal,  state  and  local  funding  sources. 

-Determine  whether  the  program  charges  clients  for  services 
rendered  and  the  extent  of  program's  use  of  third  party  pay- 
ment sources. 
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-Determine  whether  annual  financial  audits  are  conducted  by  a 
certified  public  accountant. 

During  this  interview  the  fiscal  officer  should  be  told  of  the 
purpose  of  the  review  and  of  the  need  to  assess  fiscal  records.   If 
appropriate,  a  fiscal  or  clerical  employee  should  be  identified  to 
assist  the  evaluator  in  locating  pertinent  files.   Following  the 
interview  with  the  chief  fiscal  officer,  the  appropriate  fiscal 
personnel  identified  by  the  fiscal  officer  should  be  interviewed  to 
provide  the  evaluator  with  an  understanding  of  fiscal  procedures 
and  to  identify  the  location  of  verification  materials  for  per- 
formance of  the  following  evaluation  procedures. 

CURRENT,  HISTORICAL  AND  FUTURE  FINANCIAL  CONDITIONS: 

Objectives  of  Review: 

The  objectives  of  this  section  of  the  evaluation  are  to  evaluate 
and  develop  conclusions  regarding  the  program's  long-range  financial 
planning  strategies  for  ensuring  the  survival  and  continuity  of  the 
program.   This  will  be  accomplished  in  part  by  evaluating  the 
historical  and  current  program  financial  condition. 

Evaluation  Procedures: 


General  Financial  Planning  Resources  and  Activities: 

To  assess  the  program's  general  approach  to  financial  planning, 
interview  the  program  director  and/or  the  chief  fiscal  officer 
and  determine  the  following: 

a.  Who  is  responsible  for  financial  planning? 

b.  What  reports  are  prepared  by  the  accounting  department 
and  how  frequently? 

c.  What  are  the  major  sources  of  income  and  expenditures  for 
the  program?   (See  the  "Estimated  Income  and  Expense 
Worksheet,"  Exhibit  III-l.) 
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EXHIBIT  III-l 
Program  XYZ 
I'^sL  imatL'd  Income  .iinJ  ['Expense  Workslic-c-L 

LAST  THREE  YEARS 


ANNUALIZED 
CURRENT 
YEAR 


INCOME 


State  (List) 

Federal  (List) 

Third  Party  (List) 

Client  Fees 

Food  Stamps 

Local  Government  (List) 

Other  (List) 

TOTAL 


EXPENSE 


Salaries  &  Fringes 
Other  Direct  Expenses^ 
Subcontractor  Services 
(If  Applicable  List  Each) 

TOTAL 


Excess  of  Income  Over 
Expenses  (If  Applicable) 

Excess  of  Expenses  Over 
Income  (If  Applicable) 


NOTE:    This  data  is  intended  to  assist  in  a  management  assessment  of 
overall  financial  condition.   It  is  not  intended  to  take  the 
place  of  an  audited  financial  statement.   Therefore,  the  eval- 
uator  should  rely  where  necessary  on  program  provided  data. 


1    Includes  food,  supplies,  rent,  travel,  etc, 
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d.  Has  the  program  identified  any  funding  source  to  sub- 
stantially supplement  federal  or  state  funding? 

e.  Has  the  program  enacted  a  client  fee  system  and  a  third 
party  painnent  collection  system? 

Health  services  funding  regulations  (42  CFR  Part  50, 
Subpart  A)  require  that  programs  receiving  federal  funds 
submit  a  three  year  plan  for  third  party  payments. 
Review  this  plan  against  the  programs  actual  third  party 
experience  and  comment  if  appropriate. 

f.  Has  the  program  considered  the  following  sources  of  third 
party  funding? 

Title  XX 

Medicaid 

Blue  Cross/Blue  Shield 

Other  Private  Insurance 

Champus 

Other 

g.  If  the  program  has  not  considered  any  of  the  above  third 
party  sources,  probe  regarding  why  not. 

h.    If  they  have  considered  these  third  party  sources  but  not 
actively  pursued  them,  probe  regarding  reasons. 
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2 .    Current,  Historical  Financial  Condition 

To  evaluate  and  draw  conclusions  regarding  current  and  histori- 
cal financial  condition  of  the  program,  obtain  the  assistance 
of  the  chief  fiscal  officer  and  review  present  and  past  revenue 
and  expenditure  documentation  and  perform  the  following  steps: 

a.  Develop  a  three  year  trend  analysis  of  the  program's 
income  and  expenditures  (if  possible) .   Document  results 
on  "Estimated  Income  and  Expense  Worksheet,"  Exhibit  III- 
1. 

b.  Evaluate  and  understand  any  major  changes  in  the  income 
and  expenditures. 

c.  Discuss  with  program  management  the  expected  financial 
condition  of  the  program  one  or  two  years  from  now. 
Develop  an  understanding  with  them  on  how  they  will 
adjust  their  program  if  any  major  changes  are  expected  in 
increased  expenditure  requirements  such  as  needed  capital  items 
or  reduced  revenues  such  as  conclusion  of  federal  grants 

or  contracts. 

d.  Subsequent  to  reviewing  the  initial  results  of  the 
overall  program  evaluation,  discuss  v/ith  management  where 
long  range  cost  savings  or  increased  revenue  might  be 
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realized  and  the  potential  impact  on  the  program.   Some 
areas  to  consider  would  be: 

.Personnel  savings. 

.Increases  in  client  fees  or  third  party  collections. 

•Increased  efficiency  in  use  of  community  support 
services  provided  at  no  cost  to  the  program. 

BUDGETING  PERFORMANCE  AND  CASH  MANAGEMENT: 

Objectives  of  Review: 

The  objectives  of  this  section  of  the  evaluation  are  to  review  the 
program's  budgeting  procedures  and  evaluate  how  actual  program 
expenditures  and  revenue  compare  with  those  budgeted,  and  to  de- 
velop conclusions  regarding  the  nature  of  any  major  variances.   A 
second  objective  is  to  evaluate  how  cash  requirements  are  developed 
and  met,  and  how  surpluses  or  deficits  are  managed  and  to  draw 
conclusions  about  this  process. 

Evaluation  Procedures: 

1 .    Current  Budgeting  Performance: 

To  assess  current  budgeting  performance  and  methods,  interview 
the  chief  fiscal  officer  and  perform  the  following  steps: 

a.    Obtain  a  copy  of  the  budget  for  the  current  fiscal  year. 
Determine  that  the  budget: 
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.Includes  line  items  for  all  the  accounts  which  will 
actually  be  used  during  the  year. 

.Is  consistent  with  the  line  item  distribution  pre- 
sented on  the  budget  submitted  to  the  Department  of 
Institutions. 

b.  Have  there  been  any  revisions  in  the  line  item  budgets? 
If  yes,  document  the  procedures  for  these  revisions. 

c.  Develop  a  comparison  of  the  program's  budgeted  versus  actual 
expenditures  (6  or  more  most  recent  months  if  possible) . 
Complete  the  "Budget  versus  Actual  Expenditure  Worksheet", 
Exhibit  III-2. 

d.  Obtain  an  explanation  for  any  major  budget  overruns 
during  the  last  six  or  more  months  and  management  steps 
taken  to  remedy  these  budget  overruns. 

2.    Cash  Management: 

To  evaluate  how  cash  requirements  are  developed  and  met,  and 
how  surpluses  or  deficits  are  managed,  and  to  draw  conclusions 
about  this  process,  interview  the  chief  fiscal  officer  and 
determine  the  following: 

a.    VJho  is  responsible  for  budgeting  requirements? 
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EXHIBIT  I I 1-2 

Program  XYZ 

Budget  Versus  Actual  Expenditures 

(X  Number  of  Months  Ended  ,  197   ) 


BUDGET 


ACTUAL 


VARIANCE 


INCOME 

State  (List) 

Federal  (List) 

Third  Party  (List) 

Client  Fees 

Food  Stamps 

State  (List) 

Local  Government  (List) 

Other  (List) 


TOTAL 


EXPENDITURES 


Salaries  &  Fringes 
Other  Direct  Expenses^ 
Subcontractor  Services 
(If  Applicable  List  Each) 

TOTAL 

Excess  of  Income  Over 
Expenses  (If  Applicable) 

Excess  of  Expenses  Over 
Income  (If  Applicable) 


NOTE:    This  data  is  intended  to  assist  in  a  management  assessment  of 
overall  budget  performance.   It  is  not  intended  to  take  the 
place  of  an  audited  financial  statement.   Therefore,  the  eval- 
uator  should  rely  where  necessary  on  program  supplied  data. 


1    Includes  food,  supplies,  rent,  travel,  etc, 
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b.    How  soon  before  the  beginning  of  the  fiscal  year  is  the 
budget  prepared? 


c.  How  are  estimates  done?   Is  the  process  formal  and  docu- 
mented? 

d.  What  happens  when  a  delay  or  curtailment  occurs  in  re- 
ceiving revenue  from  a  particular  funding  source? 

e.  How  does  the  program  finance  its  current  expenses  in  this 
situation?   (Probe) 

f .  Does  the  program  accelerate  its  draw  down  of  federal  or 
state  funds  when  necessary? 

g.  Does  the  program  utilize  (or  plan  to  utilize)  its  un- 
obligated federal  or  state  balances  (if  any)  for  these 
purposes? 

h.    Has  the  program  developed  a  surplus  for  working  capital? 

i.    For  mental  health  programs,  perform  the  following  steps 
to  ensure  that  available  non-state  funds  are  spent  prior 
to  state  funds,  and  that  cash  reserve  accounts  do  not 
exceed  17%  of  a  mental  health  center's  total  annual 
budget : 
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, To  ensure  that  "available"  cash  reserve  accounts  do 
not  exceed  17%  of  a  mental  health  center's  total 
annual  budget: 

-Obtain  the  Department  of  Institutions'  "Defini- 
tion of  Funds  Not  Available"  policy  memorandum 
dated  August  3,  1976.   Determine  how  the  CMHC 
has  defined  its  "available"  and  "non-available" 
cash  accounts,  i.e.,  operating  cash,  endowment 
funds,  capital  reserves,  cash  reserves,  etc.   A 
governing  board  resolution  supporting  the 
definitions  should  be  obtained  and  reviewed. 
The  definitions  should  be  in  compliance  with 
the  "Definition  of  Funds  Not  Available"  memoran- 
dxim  issued  by  the  Department  of  Institutions. 

-Obtain  a  total-center  balance  sheet  (preferably 
from  audited  financial  statements)  for  the  most 
recent  fiscal  year-end.   Divide  the  total 
amount  shown  for  "available  cash  reserve" 
accounts  by  the  total  amount  of  operating 
expenses  budgeted  for  the  current  fiscal  year. 
If  this  ratio  exceeds  17%,  the  CMHC  is  not  in 
compliance. 

.To  ensure  that  "available"  funds  other  than  state 
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general  fund  monies  are  spent  prior  to  state  general 
fund  monies: 

-Review  the  total  center  budget  for  the  current 
fiscal  year.   If  budgeted  income  is  greater 
than  budgeted  expenses,  determine  from  the 
fiscal  officer  the  reason  for  the  excess,  and 
whether  that  excess  will  be  placed  in  a  "non- 
available"  cash  account. 

NOTE:      If  the  excess  is  to  be  "avail- 
able", the  center  is  a  priori 
out  of  compliance,  and  the  state 
grant  should  be  correspondingly 
reduced. 

-Review  actual  income  and  expenses  for  the 
current  fisal  year-to-date  versus  the  budgeted 
year-to-date. 

— If  actual  expenses  are  under  budget, 
determine  whether  claims  or  receipts  of 
state  grant  funds  are  under  budget  by  the 
same  amount  (see  the  state  grant  income 
line  item).   If  they  are  not,  state  general 
funds  are  not  being  spent  last. 
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— If  actual  income  in  any  category  other 
than  state  grant  income  is  greater  than 
budget  and  is  not  matched  by  corresponding 
actual  expense  increases,  determine  whether 
the  excess  income  is  placed  in  an  "avail- 
able" cash  account.   If  it  is,  then  deter- 
mine whether  claims  or  receipts  of  state 
grant  funds  are  under  budget  by  the  same 
amount.   If  they  are  not,  state  general 
funds  are  not  being  spent  last. 


D.    IMPLEMENTATION  OF  INTERNAL  FINANCIAL  CONTROLS: 

For  the  following  evaluation  procedures,  the  evaluator  should  work 
closely  with  the  chief  fiscal  officer  and  his  staff  to  complete  the 
evaluation  steps.   In  the  first  section  (determination  of  the 
system  of  internal  control) ,  the  evaluator  should  interview  the 
chief  fiscal  officer  and  complete  the  "Internal  Financial  Controls 
Questionnaire",  Exhibit  III-3. 
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EXHIBIT  III-3 

INTERNAL  FINANCIAL  CONTROLS 

QUESTIONNAIRE 


E. 


INTERNAL  FINANCIAL  CONTROLS: 


This  section  is  organized  by  the  following  three  subsections: 

-Determination  of  the  system  of  Internal  control. 

-Test  of  transactions  for  verification  of  selected  controls. 

-Summary  of  the  financial  systems'  strengths  and  weaknesses. 

Where  appropriate,  each  subsection  is  prefaced  by  a  statement  of 
review  objectives  and  the  "ideal  conditions"  the  Department  of 
Institutions  believes  should  be  present  for  all  community-based 
program  accounting  systems.   For  the  following  evaluation  pro- 
cedures, the  evaluator  should  work  closely  with  the  chief  fiscal 
officer  and  his  staff  to  complete  the  evaluation  steps.   In  the 
first  section  (determination  of  the  system  of  internal  control)  , 
the  evaluator  should  interview  the  chief  fiscal  officer  and  com- 
plete the  following  questionnaire. 

-General  Accounting  Issues: 

Review  Objective: 

To  develop  an  overview  of  the  accounting  system  and  develop  a 
work  plan  for  the  remainder  of  the  evaluation. 

Ideal  Conditions: 

Wherever  possible,  there  is  separation  of  functions  and  duties 
such  that  no  employee  has  the  ability  or  originate  and  complete 
a  transaction  without  adequate  checks  and  review  by  responsible 
officials.   Accounting  records  are  complete  and  current. 
There  are  written  descriptions  of  the  accounting  policies  and 
procedures  in  effect.   In  all  respects  they  are  a  useful  tool 
with  which  management  may  measure  the  progress  of  the  program. 


1 .    General  Ledger,  Books  of  Original  Entry,  and 
Journal  Entries: 

a.  General  ledger  accounts  are  balanced  monthly 
and  reviewed  by  a  responsible  official. 

b.  All  subsidiary  ledgers  and  books  of  original 
entry  are  balanced  monthly  and  agreed  to  the 
general  ledger. 

1)    By  personnel  independent  of  preparing 
ledgers/books  of  original  entry. 

c.  All  journal  entries  require  the  approval 

of  a  responsible  officer  who  is  not  involved 
in  the  preparation  of  the  journal  entries  or 
initiation  of  entries  in  the  books  of  original 
entry. 


EMPLOYEE 

PERFORMING 
PROCEDURE 

PROCEDURE 
EMPLOYED 

N/A 

YES 

NO 
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d.  The  general  accounting/general  ledger/ journal 
entry  function  is  completely  separate  from 
custody  and  control  over  assets  such  as 

cash  and  inventory. 

1)    Personnel  performing  this  function  are 
not  involved  in  the  detail  recording 
functions  of  accounts  receivable, 
accounts  payable,  or  purchasing. 

e.  NOTE:   In  those  situations  where  no  formal 
G/L  is  in  use  but  the  organization  uses  a 
modification  thereof  as  its  source  journal, 
proceed  as  if  this  were  a  formal  G/L. 

2 .  Accounting  Administration: 

a.  A  complete  and  current  chart  of  accounts 
is  in  existence,  which 

1)    includes  descriptions  of  items  which 
go  into  each  account. 

b.  A  complete  and  current  accounting  manual 
exists,  includes  job/responsibility  descrip- 
tions and  is  used  by  accounting. 

c.  There  are  adequate  physical  safeguards 
maintained  over  accounting  records,  e.g., 
books  of  original  entry,  the  general  ledger, 
etc. 

d.  Access  to  accounting  records  is  limited 

to  employees  with  designated  responsibility 
for  such  records. 

e.  Reasonable  record  retention  and  filing 
procedures  exist. 

1)    Condition  of  accounting  records/files 
is  proper  and  orderly. 

f.  Unallowable  costs  (costs  which  sponsoring 
federal  or  state  government  agencies  do  not 
allow  as  charges  to  their  grants  and  con- 
tracts) are  clearly  identified  and  segre- 
gated in  the  accounting  records. 

3.  General  Matters  Regarding  Personnel: 

a.  All  personnel  in  a  position  of  trust  are 
bonded.   ($25,000) 

b.  Employees  in  accounting  are  experienced 
and/or  adequately  trained  for  their  duties. 

1)  Turnover  is  not  so  severe  as  to  pro- 
hibit an  efficient  accounting  operation. 

2)  There  is  internal  training  to  prepare 
employees  to  perform  their  duties 
ofllclently  and  to  fill  in  for  othiTS 
when  absences  occur. 

c.  All  employees  in  a  position  of  trust  must 
take  annual  vacations  during  which  their 
duties  are  performed  by  other  personnel. 


EMPLOYEE 

PERFORMING 

PROCEDURE 


PROCEDURE 
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N/A   YES    NO 
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d.  The  program  maintains  a  specific  policy  on 
conflict  of  interest.   The  policy  specifies 
that  personnel  in  a  position  of  trust  are  not 
related  to  each  other  and  employees  are  pro- 
hibited from  having  business  dealings  with 
companies  affiliated  with  or  acting  as  major 
suppliers  of  the  program. 

e.  All  advances  to  employees  are  adequately 
supported  by  vouchers,  approved  by  a  re- 
sponsible official.  (NOTE:  If  advances 
are  made,  how  are  they  funded?) 

f.  Travel  and  expense  accounts  are  adequately 
supported  and  are  reviewed  by  a  responsible 
official  for  compliance  with  program  policies 
prior  to  reimbursement. 

Miscellaneous: 

a.  A  tickler  file  is  maintained  of  deadline 
dates  for  reporting  requirements,  e.g., 
taxes,  federal  government,  ASD,  etc. 

b.  A  memo  record  is  maintained  of  all  dormant 
bank  accounts. 

c.  Adequacy  of  insurance  coverage  on  all  assets 
is  frequently  reviewed  by  an  insurance  expert. 

External  Auditing: 

a.  Is  an  external  audit  performed? 
YES  NO  

b.  If  yes,  how  often  is  an  audit  performed? 


c . 
d. 


By  whom? 


Obtain  a  copy  of  the  most  recent  audit  report. 

Did  the  audit  specify  whether  there  is  evi- 
dence of  inadequate  internal  fiscal  control? 
If  yes,  obtain  documentation. 


EMPLOYEE 
PERFORMING 
PROCEDURE 


PROCEDURE 

EMPLOYED 


N/A   YES    NO 
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-Payroll  and  Personnel: 

Review  Objective: 

To  determine  if  adequate  procedures  exist  to  properly  process 
payroll  given  the  size  and  complexity  of  the  organization. 

Ideal  Conditions : 

New  employees  are  investigated  before  being  hired.   Wage  and 
personnel  policies  are  current  in  writing.   Current  and  com- 
plete personnel  files  are  maintained.   An  adequate  system  is 
used  tc  insure  proper  recording  of  hours  and  time,  e.g., 
weekly  time  reports  are  being  used  for  all  employees.   All 
salary  and  wage  payments  are  made  by  checks  which  are  signed 
and  distributed  by  persons  other  than  those  preparing  the 
payroll  or  controlling  hiring  and  termination.   All  payroll 
computations  are  independently  double  checked.   The  payroll 
bank  account  is  reconciled  by  someone  other  than  the  person 
preparing  the  payroll  or  signing  the  checks. 


1.  Personnel,  Employment  and  Rate  Authorizations: 
a.    An  appropriate  official  authorizes: 

1)  Initial  rates  of  pay. 

2)  Any  subsequent  changes  in  rates  of 
pay  or  position. 

2 .  Payroll  Preparation  and  Time  Keeping: 

a.  Time  keeping  and  time  cards: 

1)  Time  keeping  is  separated  from 
payroll  preparation. 

2)  Formal  attendance  records  are 
used,  e.g.,  time  clock  cards, 
weekly  time  reports,  etc. 

3)  Time  keepers  use  face  check  sheets 
or  similar  methods  to  check  atten- 
dance where  there  are  no  time 
clocks. 

4)  Where  time  clock  cards  are  in  use, 
controls  exist  to  prevent  one  per- 
son from  punching  more  than  one 
time  card . 

5)  Approval  of  department  head  on 
time  cards  or  other  attendance  data 
is  required  prior  to  preparing 
payroll . 

6)  Changes  on  time  cards  must  be 
initialed/approved  by  a  depart- 
ment head  or  similar  person. 

b.  Persons  preparing  the  payroll  do  not 
perform  other  payroll  duties  (e.g.', 
time  keeping,  distribution  of  checks) 
or  have  access  to  other  payroll  data 
or  cash. 


EMPLOYEE 

PERFORMING 

PROCEDURE 


PRODEDURE 
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c.  Total  payroll  Is  checked  to  labor 
distribution  totals  independently 
determined  In  other  departments  from 
original  time  cards. 

d.  Payroll  performs  the  following  func- 
tions in  preparing  payroll: 

1)  Checks  time  card/attendance  records 
for  computations  of  payroll  period 
hours . 

2)  Reviews  time  card  records  for 
specific  overtime  approval  as  well 
as  approvals. 

3)  Checks  overtime  hours,  rates,  and 
computations. 

4)  Reviews  time  records  for  unapproved 
erasures  and  alterations. 

5)  Verifies  pay  rates. 

e.  All  clerical  operations  in  payroll  prepara- 
tion are  double  checked  before  payment  is 
made . 

f.  Payroll  is  subject  to  final  approval  before 
payment  by  someone  independent  of  payroll 
preparation  and  time  keeping.   Who  does  this? 

g.  Where  machine-signed,  there  are  adequate 
controls  over  the  number  of  uses  of  the  sig- 
nature plate. 

h.    Payroll  checks  are  prenumbered,  blank  stock 
is  controlled,  checks  are  used  in  numerical 
sequence,  and  numerical  sequence  is  accounted 
for  and  reconciled  to  payroll  check  registers. 

i.    All  voided/spoiled  checks  are  properly 
mutilated  and  retained. 

i .    Checks  contain  a  detail  of  gross  pay  and 
deductions. 

k.  Monthly,  quarterly,  or  annually  there  is  a 
reconciliation  of  gross  and  net  pay  amounts 
as  shown  on  tax  returns  to  total  payroll  on 
tax  returns  to  total  payroll  on  the  payroll 
register  and  general  ledger.  NOTE:  Be  sure 
to  check  timely  filing  of  Federal  and  state 
forms  (940,  941,  etc.). 

3.    Payment  of  Payroll: 

a.    Employees  are  paid  by  checks,  which  are: 

1)  Prepared  by  persons  independent  of 
time  keeping. 

2)  Signed  by  persons  not  preparing  checks 
or  pay  registers. 
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b.  If  employees  are  paid  in  cash: 

1)  Person  preparing  pay  envelopes  is  not 
involved  in  preparing  payroll  or  time 
keeping. 

2)  Cash  is  inserted  in  envelopes  or  the 
payrolls. 

c.  Pay  checks/envelopes  are  distributed  by 
someone  independent  of  time  keeping  or 
preparing  payroll,  checks,  or  envelopes. 

d.  Paymasters  function  Independently  of  payroll 
operations,  are  rotated  periodically,  and 
are  accompanied  by  a  person  who  is  not  in- 
volved in  payroll  preparation  when  delivering 
checks. 

e.  Signed  payroll  receipts  are  obtained  from 
all  employees  when  they  are  paid  in  cash. 

1)   All  pay  receipts  are  cancelled  promptly, 
after  proof  and  reconcilement  against 
disbursements,  to  prevent  unauthorized 
reuse  of  such  receipts. 

f .  Payroll  is  paid  from  an  imprest  bank  account 
restricted  to  payroll. 

Unclaimed  Pay  Envelopes/Checks: 

a.  Are  returned  to  an  employee  disassociated 
with  the  payroll  function. 

b.  Are  released  only  on  presentation  of  proper 
identification  by  employee  and  preparation 
of  a  signed  receipt. 

c.  A  continuing  record  is  maintained  of  all 
unclaimed  wages. 

Year-End  Preparation  of  W-2s: 

a.  Total  of  W-2  wages  for  the  year  is  reconciled 
to  general  ledger  and  payroll  register  wages 
paid. 

b.  W-2s  returned  or  unclaimed  are  received  and 
investigated  by  other  than  payroll  and  time 
keeping  personnel. 

Payroll  Prepared  with  EDP  Equipment: 

a.  Master  file  access  is  restricted.  Identify 
who  has  access. 

b.  Total  hours,  employees,  etc.  are  balanced 
and  reconciled  monthly. 

c.  Reports  of  all  master  file  changes  are 
received  and  reviewed  by  personnel  independent 
of  payroll  preparation  and  time  keeping. 

d.  Adequate  control  exists  on  use  of  signature 
plates. 
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e.    Adequate  physical  and  accounting  controls 
exist  over  blank  checks  while  in  EDP  area. 

7 .    Reconciliation  of  Payroll  Bank  Accounts  and 
General: 

a.  Payroll  bank  accounts  are  reconciled  by 
employees  who  have  no  other  payroll  functions. 

b.  Reconciliation  procedures  include  all  pro- 
cedures specified  for  reconciliation  of 
normal  commercial  disbursing  accounts. 

c.  Salary  and  special  payrolls  are  subject  to 
the  same  critical  review  as  regular  payroll. 

d.  Hourly  payrolls  are  compared  as  between 
periods  and  adequate  explanation  is  required 
for  significant  variances. 

e.  The  payroll  distribution  entry  is  prepared 
by  someone  having  no  responsibility  for 
payroll  preparation,  or  preparation  or  dis- 
tribution of  payroll  envelopes  or  checks. 

f.  If  pension  costs  and/or  vacation  accruals 
are  involved,  there  are  controls  to  assure 
adequate  segregation  of  payroll  costs  appli- 
cable to  the  computations  of  those  amounts, 
including  control  over  eligible  employees 
and  changes  in  qualifications  and  rates. 


PROCEDURE 
EMPLOYED 


N/A   YES    NO 
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-Procurement  and  Payments: 

Review  Objective: 

To  review  and  develop  conclusions  about  the  programs  overall 
purchasing  and  cash  disbursements  functions. 

Ideal  Conditions: 

Purchasing  and  Accounts  Payable: 

All  significant  purchases  must  be  supported  by  an  ap- 
proved purchase  order.   Where  warranted,  dollar  limita- 
tions on  purchases  without  officer  or  board  approval  are 
printed  on  the  face  of  the  purchase  order  form.   Major 
vendors  should  be  advised  of  purchasing  requirements  and 
policies.   Competitive  bids  are  required  on  purchases 
over  a  specified  amount.   Numerical  and  physical  control 
is  maintained  over  unissued  purchase  order  stock.   Pur- 
chase orders  are  used  in  numerical  sequence.   The  pur- 
chasing system  is  designed  to  insure  accurate  and  prompt 
recording  or  purchases,  payables  and  the  taking  of  all 
discounts.   Copies  of  purchase  orders  are  sent  as  issued 
to  receiving  and  accounting.   Purchasing  personnel  are 
not  involved  in  receiving,  payables,  disbursing  or  other 
conflicting  functions.   Vendors'  invoices  are  checked  to 
purchase  orders  and  receiving  reports  and  clerical  ac- 
curacy is  proven  before  approval  for  payment.   Vendors' 
statements  are  checked  to  recorded  liabilities  regularly 
by  independent  personnel. 

Cash  Disbursements: 

Prenumbered  voucher  checks  are  used  on  all  disbursements. 
Numerical  and  physical  control  is  maintained  over  blank 
as  well  as  issued  checks.   Authorized  signatures  are 
approved  by  the  Board  of  Directors  and  are  periodically 
reviewed  and  updated.   Counter  signatures  are  required  on 
all  checks  over  a  specified  amount.   A  check  protector  is 
used  and  check  signing  machines  and  facsimile  signature 
plates  are  properly  controlled.  All  supporting  documents 
are  presented  to  authorized  signers  along  with  check  for 
signature  and  are  cancelled  at  the  time  of  signature. 
Checks  are  adequately  safeguarded  before  mailing  and  are 
mailed  by  someone  independent  of  disbursing.   Voided  or 
spoiled  checks  are  adequately  mutilated  and  are  retained 
on  file  in  sequence.   Disbursing  personnel  are  not  in- 
volved in  receiving,  purchasing,  or  receipts. 


I .   Purchasing: 

a.  Purchase  orders  are  used  on  all  purchases 
and  they  show  vendor,  date  price,  quantity 
and  delivery  terms,  method,  date  and  desti- 
nation. 

b.  Purchasing  function  is  performed  by  and 
purchase  orders  are  prepared  by  personnel 
independent  of: 

1)  Receiving  and  shipping  functions. 

2)  Payables  and  disbursing  functions. 
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Special  limitations  exist  on  and  approvals 
are  required  for: 

1)  Purchases  for  or  from  employees. 

2)  Purchases  of  equipment. 

3)  Purchases  of  goods  delivered  directly 
to  clients. 

A  current  and  adequate  manual  of  purchasing 
procedures  exists.   (The  reviewer  should  re- 
view it  in  conjunction  with  internal  control 
in  this  area.) 

Copies  of  purchase  orders  are  distributed  to: 

1)  Vendor 

2)  Purchasing  department  historical  vendor 
files. 

3)  Accounting. 

4)  Receiving. 

5)  Others   (List)  


f.  Purchase  orders  are: 

1)  I'rt,rniintl><.TC*d  and  used  in  scqiicncf . 

2)  Initiated  on  the  basis  of  purchase 
requisitions  approved  by  a  responsible 
employee. 

3)  Limited  to  purchases  less  than 

$ on  their  face  without  prior 

approval  by  an  officer  or  the  Board  of 
Directors. 

4)  Prior  to  mailing,  approved  by  a  respon- 
sible employee  not  associated  with  re- 
ceiving, payables  or  disbursements. 

g.  Numerical  sequence  is  checked  by  an  employee 
independent  of  initiation  of  purchases. 

Accounts  Payable: 

a.  Vendors  invoices  and  all  copies  are: 

1)  Routed  directly  to  accounting  from 
mail  room. 

2)  Stamped  so  that  all  copies  are  clearly 
marked  "DUPLICATE". 

b.  Original  vendors  invoices  are: 

1)  Vouchered  and  listed/recorded  in  a 
voucher/purchases  register  immediately 
on  receipt. 

2)  Maintained  in  the  accounting  department 
and  only  copies  are  distributed  when 
required. 
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c.  Vouchers  prenurabered  and  numerical  sequence 
is  periodically  checked  by  an  employee  in- 
dependent of  voucher  preparation. 

d.  In  processing  the  invoice  and  related 
voucher: 

1)  All  data  thereon  is  checked  to  approved 
copies  of  the  purchase  order  and 
receiving  report. 

2)  All  extensions,  footings,  discounts 
and  freight  terms  are  checked  for 
accuracy  as  required  by  program  policy. 

3)  The  per  rorm;incc  of  1)  and  2)  is  cle.Trly 
indicated  on  tlie  face  of  the  invoice. 

4)  A  responsible  employee  reviews  and 
approves  the  invoice  account  distri- 
bution and  approves  for  payment. 

5)  The  total  of  unprocessed  vendors  In- 
voices and  vouchered  and  unpaid  items 

is  determined  and  reconciled  to  voucher/ 
purchases  register  and  accounts  payable 
detail  at  month  end  by  someone  inde- 
pendent of  the  detail  posting  function. 

e.  Vendors'  month  end  statements: 

1)  Are  reconciled  periodically  by  an 
employee  Independent  of  voucher  prepa- 
ration to  recorded  liabilities  based 
upon  Invoices  accrued. 

2)  Are  marked  "statement"  to  prevent 
payment  if  the  program  pays  on  an 
Invoice  basis. 

3)  Are  reviewed  for  non-current  invoice 
dates. 

f.  If  advance  payments  are  made  to  vendors: 

1)  They  are  regularly  approved  prior  to 
payment. 

2)  They  are  set  up  as  accounts  receivable 
or  equivalent. 

g.  A  record  is  kept  on  all  recurring  payments 
to  prevent  duplicate  or  skipped  payments, 
e.g.,  rent,  insurance,  debt,  etc. 

h.    Personnel  performing  the  payables  function 
are  not  involved  in  the  purchasing,  re- 
ceiving, disbursing  inventory  and  general 
ledger  functions. 

Cash  Disbursements: 

a.   Check  Stocks: 

1)  All  disbursements  are  made  by  check. 

a)   Are  any  expenditures  over  $100 
made  in  cash? 

2)  Checks  are  printed  on  protected  paper. 
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1)    Checks  are  pronumbered  and  used  In 
sequence . 

A)  Physical  and  accounting  controls  over 
unauthorized  use  of  blank  check  stock 
are  adequate. 

5)    There  is  a  specified  custodian  for 
blank  check  stocks. 

Check  Preparation: 

1)  Checks  are  prepared  by  specified 
employees  who  are  Independent  of 
voucher  approval. 

2)  Employees  preparing  checks  compare 
all  data  on  voucher  and  support 
checks . 

3)  All  checks,  disbursement  vouchers 
or  check  requests  are  approved  for 
payment  by  the  check  signer  or  other 
responsible  employee. 

4)  Checks  are  recorded  in  disbursements 
book  as  prepared. 

5)  Checks  are  made  payable  to  specified 
payees  and  never  to  cash  or  bearer. 

6)  A  check  protector  is  used. 

7)  Voided/spoiled  checks  are  properly 
mutilated  and  retained. 

8)  All  support  accompanies  check  when 
presented  for  signature. 

9)  All  supporting  documents  are  properly 
cancelled  at  time  of  signature  to  pre- 
vent duplicate  payment. 

Check  Signing: 

1)  Check  signers  are  authorized  by  the 
Board  of  Directors. 

2)  Authorized  signatures  are  independent 
of: 

a)  Voucher  preparation  and  approval 
for  payment. 

b)  Check  preparation,  cash  receiving 
and  petty  cash. 

c)  Purchasing  and  receiving. 


3)    All  checks  over  $ 

manual  signatures. 


require  two 


a)    When  countersigned,  both  signers 
review  vouchers  and  supporting 
data  and  are  Independent  of  each 
other. 

4)    Signing  blank  checks  is  forbidden. 
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5)    Custody  of  checks  after  and  before 
mailing,  and  mailing  of  checks  is 
handled  by  an  employee  independent 
of  all  payable,  disbursing,  cash, 
receiving,  and  general  ledger  functions. 


6) 


If  check  signing  machines  are  used, 
facsimile  signature  plates  are  adequate- 
ly controlled,  are  used  in  the  presence 
of  the  custodian,  and  are  controlled  by 
using  numbering  devices. 


d.    Bank  Reconciliations: 

1)  Bank  reconciliations  are  prepared 
monthly  on  all  accounts  by  employees 
independent  of  check  signing,  cash 
recording,  and  receiving/disbursing 
functions. 

2)  Bank  statements  are  received  unopened 
by  the  reconciler  and  physical  control 
is  maintained  over  the  statements  until 
reconciliations  are  complete. 

3)  Bank  reconciliation  procedures  Include: 

a)  Examination  of  paid  checks  for 
dates,  payees,  amounts,  cancel- 
lations and  endorsements. 

b)  Test  comparison  of  paid  checks  and 
receipts  posting  to  and  test  foot- 
ing of  books  of  original  entry. 

c)  Accounting  for  numerical  sequence 
of  paid  checks. 

d)  Verification  of  all  reconciling 
items. 

e)  Periodic  follow-up  on  old  out- 
standing checks  and  stop  payment 
notices. 

f)  Review  of  month  end  interbank 
transfers  for  propriety. 

g)  Reconciliation  and  comparison  of 
total  debits  and  credits  on  bank 
statement  to  books  of  original 
entry. 

A)        Bank  reconciliations  are  reviewed  and 
approved  by  a  responsible  employee 
promptly  upon  completion. 

NOTE:   The  lack  of  segregation  of  duties 
is  a  frequently  observed  problem  because 
of  program  size.   Temper  the  recommendations 
with  the  size  of  the  program. 


F»«?T,OYEF 

PinrEniiRK 


PRnrEniiRr 
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N/A 


Mn 
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-Supplies  Inventory: 

Review  Objective: 

To  review  whether  the  organization  has  a  system  to  monitor  and 
safeguard  their  supplies. 

Ideal  Conditions: 

A  receiving  function  separate  from  custody  of  or  access  to 
the  stock  room  is  established  which  checks  all  goods  on  re- 
ceipt.  Controls  exist  over  receipts  through  the  use  of  pre- 
numbered,  numerically  controlled  receiving  reports. 

Complete  physical  Inventories  are  taken  at  least  once  a  year 
in  accordance  with  sound  written  procedures  adequately  super- 
vised.  Physical  counts  are  reconciled  to  perpetual  records 
and  general  ledger  controls  which  are  adjusted  as  required. 
Where  used,  perpetual  Inventory  records  are  maintained  In 
balance  with  general  ledger  controls  and  are  reviewed  peri- 
odically for  unusual  Inventory  conditions,  slow  moving,  over- 
stocked items,  etc. 

Inventory  usage  and  shipments  are  controlled  by  the  use  of 
prenumbered,  numerically  controlled  stores  requisitions,  and 
shipping  tags.   Physical  facilities  for  storage  of  inventory 
nllow  for  efficient  access  to,  use  of,  and  physical  control 
over  Inventory.   Adequate  controls  exist  for  Inventory  stored 
in  outside  locations. 


1. 


Receiving: 

a.  Separate  receiving  function  exists. 

b.  Receiving  verifies  quantity  and  quality 
of  materials  on  receipt. 

c.  Formal  receiving  reports  are  utilized. 

d.  Receiving  documents  are  properly  dispersed 
throughout  the  system. 


F>fPl,ovr,F 
PF.RFnp.rUNr; 


PRnCEnURF 
F>fPT,nYFT) 


N/A_ 


Physical  Inventory: 
a 


All  goods  are  physically  inventoried  at 
least  annually. 


b.  Physical  inventories  are  supervised  by 
a  responsible  employee. 

c.  Written  Inventory  taking  procedures, 
determined  by  responsible  officials  exist. 

d.  Perpetual  inventory  records  are  reconciled 
and  adjusted  at  least  once  a  year  to  physical 
Inventory  quantities  and  such  differences 
are  Investigated  by  a  responsible  employee. 

3.    Protection  of  Inventory: 

a.    There  is  adequate  physical  protection  and 
insurance  coverage  on  inventory. 


YKS 


NO 
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1)  All  Inventory  is  stored  in  assigned 
areas  into  which  access  is  limited, 
particularly  items  of  significant 
value  susceptible  to  pilferage  (such 
as  high  value  small  items) . 

2)  Program  premises  are  adequately 
surrounded  by  appropriate  protective 
devices  such  as  fences,  guards,  door 
or  gate  checks,  etc. 


EMPLnVFE 

PERFORTTINC 

PnnCFDI'P.E 

ppncKnuRF 

F^ff'T.OVF'' 

•I/A 
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Mn 
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-Revenue  Sources,  Billing  and  Collections: 

Review  Objective: 

To  review  and  develop  conclusions  about  the  organizations 
systems  for  the  receipt  and  recording  of  revenue. 

Ideal  Conditions: 

Revenue,  Billing  and  Accounts  Receivable: 

The  program  has  identified  sources  of  funds  other  than 
federal  or  state  (social  security,  welfare,  private 
insurers,  etc.)  and  has  instituted  procedures  to  pursue 
these  sources.   There  is  adequate  separation  of  duties 
between  personnel  in  the  billing  and  accounts  receivable 
functions.   Each  entry  into  the  billing  and  accounts 
receivable  ledger  is  supported  by  adequate  evidence.   All 
non-cash  credit  entries  such  as  allowances,  bad  debts, 
etc.,  are  properly  authorized.   Subsidiary  receivable 
ledger  functions  are  performed  by  personnel  having  no 
access  to  incoming  receipts  or  credit  functions.   Sub- 
sidiary ledgers  are  balanced  and  aged  regularly. 

Cash  Receipts: 

Cash  receiving  functions  are  handled  by  as  few  people 
as  possible.   Persons  receiving  cash  have  no  access  to 
accounting  records.   Incoming  cash  receipts  are  primarily 
in  the  form  of  checks  and  drafts  and  are  deposited  daily 
and  Intact.   The  receipt  of  cash  is  provable  by  other 
separately  maintained  records.   Periodic  reconciliations 
of  detail  cash  receipt  records  to  duplicate  deposit 
slips,  accounts  receivable  and  cash  control  accounts, 
etc.,  are  prepared  by  personnel  Independent  of  cash 
functions. 


Federal  Funds: 

a.  The  grant  or  contract  has  been  signed  for 
the  current  program  fiscal  year. 

b.  The  program  has  an  indirect  cost  rate. 

c.  The  current  grant  or  contract  has  been 
adjusted  to  reflect  the  unexpended  balance 
from  prior  years. 

d.  The  program  is  drawing  down  federal  funds 
at  a  uniform  rate.   If  no,  why?  


Third  Party  Fees: 

a.  An  ongoing  procedure  exists  for  obtaining 
and  updating  information  on  third  party 
coverage  and  eligibility  status  of  project 
clients. 

b.  The  program  maintains  records  on  third 
party  coverage  and  on  the  ability  of  pa- 
tients to  pay  for  services  where  applicable. 

c.  Written  agreements  exist  between  the  program 
and  all  third  party  payors  or  patients  which 
specify  the  services  to  be  provided  and  fees 
to  be  charged. 


r^TPLnYEF 

PFRFORMTMC 
PROCEmiRR 


N/A 


YES 
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d.   Third  party  billings  are  based  upon  an 
estimate  of  actual  program  operating 
experience,  e.g.,  the  program  has  attempted 
to  identify  the  actual  costs  of  treatment  for 
which  it  is  billing.   (Review  and  document 
their  procedures.) 

Client  Fee  System: 

a.  The  program  has  enacted  a  client  fee 
schedule. 

b.  The  client  fee  schedule  is  based  upon  an 
estimate  of  actual  program  operating 
experience. 

c.  Are  all  services  billed?  Are  they  recorded 
as  receivables?  Are  they  attempting  to 
collect  these  fees? 

Local  Money: 

a.  Formal,  signed  contracts  are  in  effect. 

b.  The  local  funds  have  special  restrictions. 

If  so,  what  are  they?   (Probe)   


The  program  has  a  separate  accounting  system 
for  state  and/or  local  funds. 

The  local  funds  are  being  received  at  a 
uniform  rate.   If  not,  probe.  


e.  The  local  funds  are  used  as  matching  funds. 

f.  Do  local  contracts  have  cancellation  provi- 
sions? 

g.  Are  any  of  the  local  contracts  scheduled  to 
expire? 

Non-Cash  Revenue: 

a.  Does  the  program  utilize  non-cash  Items 
(such  as  food,  furniture  and  volunteers)  as 
matching  funds  for  federal  or  state  funds? 

b.  Is  there  any  consistency  to  the  receipt 
of  non-cash  revenue? 

c.  Are  the  receipt  of  these  items  documented? 

d.  Do  they  have  a  formalized  system  for  assign- 
ing a  monetary  value  to  such  things  as  used 
furniture,  building  space  and  volunteer  time? 

e.  Are  items  which  are  claimed  as  match  during 
the  current  fiscal  year  actually  used  in  the 
operations  of  the  program?   Is  such  use  for- 
mally documented? 

Billing: 

a.    Written  policies  and  procedures  exist  for 
billing  of  services. 


EOTI.nYEE 

PERFORMTNG 

PRnCEDT'RF 


PROCKnURE 


N/A 
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NO 
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h.    The  billing  department  mails  invoices 

directly  with  a  copy  going  to  the  accounts 
receivable  function  to  be  used  for  posting 
receivable  ledgers. 

c.  A  system  of  follow-up  is  in  effect  whereby 
patient  service  records  are  reviewed  after 
the  fact  to  assure  that  all  service  provided 
is  billed  or  otherwise  accounted  for. 

d.  Supporting  documentation  is  maintained. 

7.  Accounts  Receivable: 

a.  Detailed  ledgers  are  maintained  for  all 
receivable  accounts.   Personnel  performing 
this  function  are  separated  from  all  cash 
functions . 

b.  Receivable  subsidiary  ledgers  are  balanced 
regularly  to  control  accounts  by  an  employee 
independent  of  detail  posting  functions. 

c.  There  is  a  clear  separation  of  duties  be- 
tween all  receivable  ledger  clerks  and  the 
cash/cashier  functions. 

d.  For  third  party,  are  monthly  statements 
sent  to  patient  and  third  party  payor? 

e.  All  entries  to  receivable  subsidiary  ledgers 
other  than  the  posting  of  cash  receipts, 
sales  invoices,  or  credit  memoranda  require 
approval  of  a  responsible  official. 

8.  Accounts  Receivable  and  Doubtful  Accounts: 

a.  Individual  accounts  in  all  receivable  ledgers 
are  periodically  aged  and  reviewed. 

b.  All  accounts  written  off  require  approval 
of  an  official  disassociated  from  detail 
receivable  and  cash  functions. 

c.  All  accounts  written  off  are  placed  in  a 
memo  control  to  prevent  their  loss  and  are 
reviewed  periodically  by  a  responsible 
official  for  potential  recovery. 

9.  Cash  Receipts: 

a.  Incoming  mail  is  opened  and  receipts  listed 
by  persons  having  no  access  to  cash  receipts 
or  accounts  receivable  records. 

1)    The  listing  of  mail  receipts  is  sub- 
sequently compared  to  cash  copies  of 
deposit  slips  by  an  employee  having  no 
access  to  cash. 

b.  Prenumbered  remittance  advices  are  prepared 
and  patient  and  third  party  remittance  data 
and  other  support  is  attached. 

c.  Cash  receipts  are  entered  in  books  of  origi- 
nal entry  by  persons  independent  of  the  mail 
opening  and  receipt  listing  function. 

d.  Receipts  are  deposited  intact  on  a  daily 
basis , 


E'-fPLOYFP, 
PERFORMING 

PRnrFniiRE 


PROCEDURE 
EMPLOYED 


N/A 
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e.  Adequate  physical  controls  exist  over  cash 
receipts  from  time  of  mall  opening  until 
time  of  bank  deposit. 

f.  Cashing  of  checks  out  of  dally  receipts  is 
forbidden. 

g.  Post-dated  checks,  disputed  Items,  unidenti- 
fied receipts,  NSF  checks,  checks  charged 
back  by  banks,  and  similar  items  are  subject 
to  normal  receipts  control,  and  received  and 
Investigated  by  persons  independent  of 
preparation  of  deposits  and  posting  of  ac- 
counts receivable  detail. 

h.    Receipts  of  affiliate  programs  or  remote 

sites  are  deposited  locally  and  are  subject 
to  central  program  withdrawal. 

1.   The  general  cashier  function  is  segregated 

from  the  general  ledger  and  subsidiary  ledger 
functions. 

j .    Persons  receiving  cash  from  over-the-counter 
and  other  sources  are  Independent  of  other 
cash  functions,  custody  of  negotiable  assets, 
posting  of  receivable  detail  and  general 
ledger. 

1)    Prenumbered  receipts  are  issued  for 

collections  and  duplicates  are  retained 
and  controlled  numerically. 

k.    Adequate  controls  exist  for  collections  made 
by  program  staff  and  others  from  patients, 
such  as  client  fees  and  food  stamps. 

1.    Program's  banks  have  been  instructed  not 
Co  cash  ctiecks  payable  to  the  program  in 
special  bank  accounts,  e.g.,  imprest  pay- 
roll accounts. 

m.    The  Board  of  Directors  authorized  all  bank 
accounts  and  check  signers. 

10.   Letter  of  Credit:    (Direct  transfer  of  money 
into  organizations  bank  account.) 

a.  The  program  submits  requests  for  payment 
on  a  timely  basis. 

b.  Is  there  a  time  lag  In  having  fund  transfers 
honored? 


EMPLOYEE 
PERFORMING 

PROCEDURE 


PROCEDURE 
EMPLOYED 
N/A   YES 


III-30 


-Petty  Cash  and  Cash  on  Hand: 

Review  Objective: 

To  determine  the  size  of  the  petty  cash  fund  and  review  wheth- 
er supporting  documentation  is  maintained  for  withdrawal . 

Ideal  Conditions: 

Petty  cash  funds  are  maintained  on  an  imprest  basis  with  one 
individual  responsible  for  the  fund.   They  are  not  commingled 
with  other  receipts.   Responsibility  for  petty  cash  Is  sepa- 
rate from  cash  disbursing/receiving  and  receivable  functions. 
Limits  exist  on  amounts  of  any  expenditure  of  or  checks  drawn 
on  petty  cash  and  on  any  unusual  expenditure,  e.g.,  wage  and 
salary  advances.   Disbursements  are  evidenced  by  approved 
support  prepared  in  ink  on  prenumbered  forms.   Support  is 
checked  and  cancelled  at  the  time  the  fund  is  reimbursed. 
Frequent  and  surprise  counts  of  funds  are  made  by  internal 
audit  or  other  Independent  personnel. 


4. 


One  employee,  separate  from  custodianship  of 
unrecorded  cash,  cash  receiving  and  disbursing 
and  accounts  or  notes  receivable  functions, 
has  custodial  responsibility  for  the  fund. 

a.    Fund  is  maintained  on  an  imprest  basis  in 
a  reasonable  amount. 

Expenditure  support  is  reviewed  prior  to  reim- 
bursement by  a  responsible  official  and  is  can- 
celled upon  reimbursement  to  prevent  reuse. 

a.    Reimbursements  are  made  payable  to  the 
custodian. 

Petty  cash  written  policies  cover  limitation  on 
amount  of  disbursement,  approval  requirements  on 
cashing  accommodation  or  personal  checks,  not  al- 
lowing cashing  of  post-dated  checks  and  placing 
restrictive  endorsements  on  checks  received. 

No  cash  receipts  from  accounts  receivable  or 

other  sources  are  commingled  with  petty  cash  funds. 

Surprise  counts  and  reconciliation  of  petty  cash 
funds  are  periodically  made  by  internal  audit  or 
other  responsible  employees. 


EMPLOYEE 

PERFORMING 

PROr.EnilRF 
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-Property,  Plant  and  Equipment: 

Ideal  Conditions: 

Subsidiary  records  exist  for  fixed  assets  and  depreciation 
reserves  which  are  reconciled  monthly  to  general  ledger 
controls  and  are  periodically  compared  to  physical  Inventories 
of  fixed  assets.   Specific  policies  on  authorization  for  and 
control  over  acquisitions,  transfers  and  sales  or  abandonments 
of  program  owned,  leased  and  contractor/government  owned/fur- 
nished assets.   Approved  depreciation  and  capitalization 
policies  exist  In  writing  and  are  adhered  to  by  accounting. 


Detail  records  or  subsidiary  ledgers  are  main- 
tained: 

a.  By  personnel  Independent  of  general  ledger, 
cash,  purchasing,  payable,  and  fixed  asset 
expenditure  authorizing  functions. 

b.  On  all  fixed  assets  and  depreciation  (In- 
cluding fully  depreciated,  and  government/ 
contractor  owned/furnished  Items). 

Actual  payments  are  approved  by  Board  of  Directors, 
if  over  $ . 

Federal  or  State  Funded  Assets: 

a.  Has  the  program  acquired  a  substantial 
number  of  assets  with  federal  or  state 
funds? 

b.  Are  these  assets  properly  tagged? 

c.  Have  physical  Improvements  been  made  to 
program  property? 


Real  Estate: 

a.  Any  owned  holdings? 

b.  How  acquired?   


c.  Is  property  mortgaged?  

d.  Is  Board  approval  required? 

e.  Identify  which  organization  holds  title  to 
the  property.  


Other  Assets: 


d. 


Does  the  Board  of  Directors  have  sole 
authority  for  authorizing  the  purchase  or 
sale  of  assets? 

Are  maintenance  logs  for  automobiles  main- 
tained? 

Do  procedures  exist  to  prevent  personal  usage 
of  organizational  vehicles? 

Are  reserves  being  established  to  ultimately 
replace  equipment  and  vehicles? 

NOTE:   Most  programs  have  few  physical  assets 
except  used  furniture.   Occasionally  a  program 
will  own  a  building  or  transportation  vehicles. 
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-Notes  Payable  and  Long-Term  Debt: 

Ideal  Conditions: 

Responsibility  for  authorization  of  long-term  borrowings  is 
vested  with  the  Board  of  Directors  or  other  management  per- 
sonnel.  Two  signatures  are  required  on  all  loan  agreements, 
evidences  of  indebtedness  and  repayment  checks.   Subsidiary 
records,  periodically  reconciled  to  general  ledger  controls, 
exist  for  all  notes  payable  and  long-term  debt  including 
related  interest.   Physical  and  accounting  controls  exist  for 
all  unissued,  issued  and  cancelled  notes  and  other  certifi- 
cates of  indebtedness.   Specified  written  procedures  exist  on 
all  aspects  of  authorization,  issuance,  and  redemption  of 
notes  and  long-term  debt  and  method  and  amount  of  paying 
Interest. 


3. 


Authorization  for  borrowings  and  execution  of 
debt  agreements  and  notes  is  vested  with  Board  of 
Directors  or  other  responsible  management  group 
personnel.   Authorizations  specifically  state: 

a.  Name  of  officers  empowered  to  borrow  funds, 
maximum  amounts  such  officers  may  borrow, 
and  maximum  terms  of  length  of  time  allow- 
able for  such  long-terra  debt. 

b.  Collateral  which  may  be  pledged  to  secure 
loans. 

c.  Allowable  loan  agreement  restrictions  or 
covenants. 

Detail  records  on  notes  and  other  long-term 
liabilities  are  maintained  by  personnel  inde- 
pendent of  cash  functions  and  are  reconciled  to 
general  ledger  controls  periodically. 

a.    On  repayment  and  receipt  of  paid  notes,  they 
nre  examined  for  proper  endorsements  and 
cancellation  and  are  returned  to  and  retained 
by  an  employee  who  does  not  maintain  the 
detail  records. 

Notes,  certificates,  and  other  evidences  of 
indebtedness  are  executed  only  in  the  program 
name  and  require  the  signature  on  such  evidences, 
as  well  as  on  principal  repayment  checks  of  two 
officials  who  are  authorized  signers  and  are 
independent  of  each  other. 


EMPLOYEE 

PERFORMING 

PROCEHURE 
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N/A   YES  NO 
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F. 


TEST  OF  TRANSACTIONS  PROCEDURES: 


General: 

The  objective  of  testing  selected  transactions  is  to  both  confirm 
the  systems  descriptions  obtained  In  the  prior  section  and  develop 
a  reasonable  level  of  confidence  in  the  cost  figures.   This  Is  not 
an  attempt  to  perform  an  audit. 

The  majority  of  program  funds  are  generally  attributable  to  per- 
sonnel costs,  and  therefore  the  tests  of  payroll  and  personnel 
should  take  priority. 

In  any  event,  the  level  of  effort  and  specific  focus  of  the  trans- 
action testing  should  be  determined  after  the  programs  overall 
system  is  understood.   Listed  below  are  indicators  which  sliould 
alert  both  the  reviewer  and  team  leader  of  a  potential  problem. 
The  focus  of  the  transaction  testing  should  be  adjusted  as  ap- 
propriate.  These  indicators  are: 

.No  audit  for  the  most  recently  completed  fiscal  year. 

.The  most  recent  audit  contained  a  disclaimer. 

.The  lack  of  staff  continuity,  particularly  the  controller  (or 
his  equivalent) . 

.The  excessive  absence  of  the  segregation  of  duties  and  func- 
tions In  the  accounting,  financial  and  administrative  areas. 

.The  program  is  under  current  litigation. 

.Excessive  delays  in  billing  and  reporting. 

.No  books  of  original  entry;  e.g.  general  ledger,  cash  dis- 
bursements, cash  receipts,  and  the  payroll  register. 

.Shortfall  in  cash. 

-Purchasing,  Accounts  Payable  and  Cash  Disbursements: 

Audit  Objectives: 

The  objective  of  test  work  in  these  areas  is  to  ascertain  that 
proper  and  timely  recording,  accrual,  and  accounting  dis- 
tribution occurs  from  the  time  an  Item  is  ordered  through  to 
when  it  is  paid  for  and  that  the  system  is  adequate  to  provide 
reasonable  assurance  against  the  perpetration  and  concealment 
of  irregularities.   To  this  end  tests  are  set  forth  below  on 
both  source  media  and  books  of  original  entry  in  the  areas  of 
purchasing,  voucherlng  and  payables,  cash  disbursing  and  bank 
reconciliations.   As  far  as  possible,  the  selection  of  data 
for  review  should  be  representative  of  transactions  found  in 
the  program's  operation  with  emphasis  on  transactions  effect- 
ing the  primary  operational  areas. 


TESTS  PROCEDURES: 

1 .  Purchasing  -  (Obtain  the  purchasing  department  purchase 
order  (PO)  file  for  the  most  recent  completed  reporting 
period  and  perform  the  following  tests:) 

a.  Account  for  numerical  sequence  of  purchase  orders 
issued. 

b.  Review  5  purchase  orders  for  propriety  of: 

1)    Completeness  in  accordance  with  program  policy, 
including  where  applicable  competitive  bid 
supporting  documents. 


N/A 

DONE 

DATE 

111-34 


N/A 


DONE 


DATE 


2)  Clerical  accuracy. 

3)  Delivery  location,  method  and  terms. 

4)  Approvals. 

I- .    *rract'  I  liusf  rOs  tv)  Liu'  d  Ishursomcnt  rilo.*^  and  chock 
to  receiving  reports,  invoices  and  otiier  support 
for  reasonableness. 

Voucher  Register  -  (If  the  program  maintains  a  voucher 
register  nr  its  equivalent,  perform  the  following  test 
for  the  most  recent  completed  reporting  period:) 

a.  Test  foot  and  crossfoot  the  voucher  register  for 
this  period  and  trace  postings  to  general  ledger. 

b.  Select  10  vouchers. 

1)  Compare  to  entries  on  voucher  register. 

2)  Compare  information  on  voucher  to  supporting 
date,  vendors  invoices,  receiving  reports, 
purchase  orders,  subcontractor  contracts,  etc. 

3)  Review  voucher  and  supporting  data  for: 

a)  Evidence  of  invoice  audit  and  approvals 
prior  to  payment. 

b)  Clerical  accuracy  on  a  test  basis. 

c)  Propriety  of  account  distribution  (deter- 
ming  that  unallowable  costs  are  properly 
segregated  from  allowable  costs  under  each 
program) . 

d)  Approvals  on  receiving  reports. 

e)  Propriety  of  prices,  quantities,  discounts 
and  freight  terms  on  invoices  in  relation 
to  related  purchase  orders. 

f)  Evidence  that  all  allowable  discounts  were 
taken. 

g)  Cancellation  of  supporting  documents  and 
voucher  after  payment. 

h)    Reasonableness  of  expenditure  in  relation 
to  nature  of  operations. 

4)  Trace  vouchers  to  and  compare  them  with  entries 
in  the  cash  disbursements  with  book  and  with 
paid  checks,  noting  dates,  payees,  signatures, 
amounts  and  endorsements. 

c.  Account  for  numerical  sequence  of  vouchers  for  the 
period. 

Cash  Disbursements 

If  the  program  has  a  separate  cash  disbursing  book/system 
obtain  the  disbursements  book  and  perform  these  tests: 

a.  Test  foot  and  crossfoot  the  cash  disbursements  book 
and  trace  postings  to  the  general  ledger. 

b.  Compare  5  cancelled  checks  for  the  period  with  the 
disbursements  book  noting  dates,  payees,  signatures, 
amounts  and  endorsements. 
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ti/A 


DONE 


DATE 


c.  Test  the  numerical  sequence  of  checks  during  the 
period  examining  void  or  spoiled  checks  for  proper 
cancellation.   (If  checks  are  still  outstanding  at 
the  time  of  the  test,  trace  them  to  the  most  current 
bank  reconciliation.) 

d.  If  the  program  maintains  a  voucher  (accrual  only) 
register,  select  the  disbursement  entries  tested  in 
b.  above  and  trace  them  to  entries  in  the  voucher 
register  and  to  the  voucher  and  supporting  data, 
reviewing  such  in  the  manner  set  for  In  2.b.3)  above, 
or 

If  the  program  does  not  maintain  a  voucher  register, 
select  10  disbursements  for  the  period  and  perform 
steps: 

2.b.l)   Compare  check  to  disbursements  book.  .  . 

2.b.2)   Compare  check  to  supporting  data.  .  . 

2.b.3)   Review  supporting  data.  .  . 

Il.ink  Kc-conc  1 1  lat  Ions 

a.    Review  reconciliation  for  period  noting  timeliness 
and  review  by  responsible  official. 

1)    Agree  balances  to  general  ledger. 
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-Payroll  and  Personnel: 

Audit  Objectives: 

The  objectives  of  test  procedures  In  this  area  are  to  as- 
certain that  existing  personnel  policies  are  being  followed, 
Ihnt  the  accounting  proicthircs  and  "system  oF  internal  control 
are  adequate  to  provide  for  timely  and  accurate  recording  and 
distribution  of  labor  costs  and  to  provide  reasonable  assur- 
ance that  payroll  payments  are  not  susceptible  to  the  occur- 
rence of  irregularities.   To  this  end,  tests  are  set  forth 
below  of  source  media,  books  of  original  entry  and  subsidiary 
ledgers  in  the  areas  of  personnel  files,  payroll  payments,  and 
labor  distribution. 


TEST  PROCEDURES: 

1 .  Payroll  Register 

Obtain  the  register  for  the  most  recent  pay  period  and 
perform  the  following  tests: 

a.  Trace  or  reconcile  totals  to  the  mor'-hly  payroll  sum- 
mary and  check  the  clerical  accuracy  of  the  summary. 

b.  Trace  the  monthly  payroll  totals  to  general  ledger. 

c.  Reconcile  the  monthly  payroll  total  to  the  monthly 
payroll  labor  cost  distribution. 

1)    Trace  labor  distribution  totals  to  general 
ledger  and  current  employee  register. 

2.  Detail  Employee  Tests 

Select  four  hourly  and  four  salaried  employees  from  the 
registers  used  in  1.  above  (See  step  7.): 

a.  Check  hours  worked  to  approved  time  cards,  time  sheets, 
etc.   Review  time  cards  for  erasures,  changes,  and 
possible  alterations  not  approved  by  the  department 
head. 

1)    Trace  hours  by  operation  or  similar  designation 
from  time  cards  to  the  detail  labor  distribution 
for  the  period,  checking  calculations  of  amounts 
charged  to  the  various  operations  or  cost  centers 
in  conjunction  with  steps  2.b.  and  2.c.  below. 

b.  Trace  hourly  or  salary  rate,  including  bonus,  overtime, 
shift  differential,  travel  time  and  vacation  pay 
wherever  applicable,  to  personnel  file  authorizations, 
approved  salary  lists. 

c.  Examine  supporting  documents  for  payroll  advances,  per 
diem,  and  any  other  similar  payment  Included  in  the 
pay  check. 

d.  Agree  exemptions  to  signed  copies  of  W-A  (See  step 
2.g.  below) . 

e.  Check  all  calculations  of  gross  pay,  tax  deductions 
and  net  pay. 

f.  Trace  all  non-statutory  deductions  (retirement,  savings 
bonds,  credit  union,  etc.)  to  authorizations  signed  by 
the  employee  in  personnel  files  or  to  other  support  and 
check  any  calculations  included  therein  (for  example, 
deduction  amount  for  retirement  fund  contribution 
based  on  length  of  service) . 


N/\ 


IWNE 


DATE 
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h. 


Pull  cancelled  checks  for  employees  selected  and 
compare  in  detail  to  payroll  register  entries, 
examining  signatures  and  endorsements  and  comparing 
payee  endorsements  to  signatures  on  W-4s. 

Compare  data  in  payroll  department  files  to  data  in 
personnel  department  master  personnel  files. 


Check  Examination 

Pull  a  block  of  25  checks  for  the  test  period  and  compare 
them  to  the  payroll  register  noting  payee,  date,  check 
number,  amounts,  and  examining  signatures  and  endorsements. 

a.  Scan  endorsements  for  unusual  or  recurring  second 
endorsements  and  investigate  any  found. 

b.  Test  the  numerical  sequence  of  checks  issued  for 
the  period  examining  all  voided  checks  for  proper 
mutilation  to  prevent  misuse. 

Terminations 

Select  two  employees  who  terminated  during  a  recent  month 
from  the  payroll  register.   (See  step  7.) 

a.  Calculate  termination  pay  in  accordance  with  program 
policies. 

b.  Review  termination  documents  in  the  payroll  depart- 
ment and  personnel  department  personnel  files  for 
propriety  of  termination  dates,  final  pay,  etc. 

c.  Review  subsequent  payrolls  to  ascertain  that  the 
employee  is  no  longer  being  paid. 

d.  Select  two  employees  who  have  terminated  during  the 
month  from  personnel  department  files.   (See  step  7.) 

1)    Agree  to  termination  data  in  payroll  department 
files  and  perform  steps  4a,  4b,  and  4c. 

Payroll  Payoff 

Consider  the  propriety  of  performing  a  payroll  payoff  for 
all  or  a  portion  of  program  personnel: 


YES 


NO 


Payoff  to  be  performed? 

Scope  of  payoff. ^ 

Approval  System 

In  conjunction  with  steps  2,  4,  and  6,  review  the  approval 
system  over  payroll  data  changes  and  also  the  system  of 
security  over  and  access  to  payroll  data.   Comment  on 
deficiencies  noted. 

Proof  of  Cash 

Review  reconciliation  for  the  most  recent  month  noting 
timeliness  and  review  by  responsible  official. 

a.    Ascertain  that  the  account  is  handled  on  a  proper 

Imprest  basis  and  that  any  intranslt  reimbursements 
are  shown  properly  as  outstanding  amounts  on  the 
reconciliation  of  the  reimbursing  account. 


H/A 


nriMP 
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-Revenue  Sources,  Billing,  Accounts  Receivable  and  Cash  Receipts: 

Audit  Objectives: 

The  objectives  of  test  procedures  in  these  areas  are  to  as- 
certain that  the  accounting  procedures  and  system  of  internal 
control  are  adequate  to  Insure  that  when  services  are  provided 
they  are  properly  billed  and  recorded,  that  resulting  receiv- 
ables are  properly  stated  and  that  appropriate  cash  collections 
are  received  and  deposited.   To  this  end,  tests  are  set  forth 
below  of  source  media,  books  of  original  entry  and  subsidiary 
ledgers  in  the  areas  of  billing,  accounts  receivables  and  cash 
receipts. 


TEST  PROCF.nURES : 

1 .  Detail  Invoice  Tests 

Obtain  the  third  party  or  patient  invoices  for  the  test 
period,  and 

a.  Select  5  Invoices  Issued  in  the  test  period  from 
account  department  invoice  files. 

1)  Trace  to  supporting  data  such  as  patient  records 
comparing  services  rendered  vs.  billed. 

2)  Review  for  appropriate  approvals,  terms,  and 
established  fee  in  accordance  with  established 
program  policies  and  fee  schedules. 

3)  Trace  in  detail  to  entries  in  the  billing  register 
and  the  accounts  receivable  subsidiary  ledger. 

b.  Verify,  by  examination  of  actual  invoices  and  by 
reference  to  billing  register  entries,  the  numerical 
sequence  of  invoices  and  credit  memos  issued  in  the 
test  period  on  a  test  basis. 

2.  General  Ledger  Accounts  Receivable  and  Billing 

Review  activity  in  the  billing  and  accounts  receivable 
accounts  in  the  general  ledger  for  one  month.   Investigate 
the  nature  of  and  review  support  for  any  entries  not  arising 
from  normal  billing  register  or  cash  receipts  journal 
sources,  particularly  non-cash  credits  to  accounts  re- 
ceivable. 

3.  Cash  Receipts  Detail  Tests 

For  one  month  during  the  test  period: 

a.  Obtain  daily  incoming  mail  cash  listings  or  their 
equivalents  and  compare  to  postings  in  the  cash 
receipts  journal. 

b.  For  a  period  of  two  days  obtain  customer  remittance 
advices.   Review  and  compare  in  detail  to  duplicate 
deposit  slips,  cash  receipts  journal,  and  to  postings 
in  detail  accounts  receivable  ledgers.   Review 
nature  of  and  support  for  miscellaneous  receipts. 

c.  Account  for  the  numerical  sequence  of  remittance 
advices  for  one  week. 

d.  Trace  total  daily  receipts  from  the  cash  receipts 
book  to  entries  on  the  bank  statement  and  investigate 
any  unusual  lags  in  date  of  deposit. 


N/A 


DOME 


DATE 
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4.   Cash  Receipts  Journal  Tests 

Obtain  the  receipts  journal  for  the  test  period: 

a.    Trace  monthly  totals  to  general  ledger  postings  on  all 
of  account  totals. 


N/A 

DONE 

DATE 
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SUMMARY  OF  FINDINGS 


Indicate  on  this  schedule  points  indicating  strengths  and  weaknesses  in  the 

program's  system  which  may  be  appropriate  for  comment  in  the  report.   These 

include  points  noted  from  using  this  document  as  well  as  from  all  other  areas 
of  work  performed  during  the  examination. 


SECT. 
REF. 

BRIEF  DESCRIPTION  OF  STRENGTHS 

BRIEF  DESCRIPTION  OF  WEAKNESS 

Attach  additional  schedules  in  this  format  as  required. 
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SECTION  IV 

CLIENT  TREATMENT  AND  CENSUS  EVALUATION  GUIDE 

(A)  MENTAL  HEALTH  PROGRAMS 

(B)  DRUG  ABUSE  TREATMENT  PROGRAMS 

(C)  ALCOHOLISM  TREATMENT  PROGRAMS 


IV. 


CLIENT  TFCEATMENT  AND  CENSUS  EVALUATION  GUIDE 


INTRODUCTION: 

Evaluation  of  the  client  treatment  system  consists  of  interviewing 
treatment  staff,  ascertaining  the  existence  of  required  treatment 
procedures  and  recording,  and  verifying  reported  treatment  data. 
The  major  elements  of  the  client  treatment  and  census  reviev;  are: 

-Treatment  process  and  services  provided. 

-Client  census  including  frequency  and  duration  of  client 

treatment  contacts. 
-Content  of  client  case  records. 
-Patient  eligibility  (mental  health  programs) . 
-Urine  Surveillance  (drug  abuse  treatment  programs) . 

Because  of  the  uniqueness  of  treatment  requirements  for  the  three 
community-based  programs,  this  evaluation  guide  is  organized  into  three 
sections  including  mental  health,  drug  abuse  treatment  and  alco- 
holism treatment  programs.   The  evaluator  should  begin  the  client 
treatment  review  with  interviews  of  appropriate  treatment  staff 
(usually  that  individual  assigned  as  the  treatment  coordinator  or 
director)  and  perform  the  following  evaluation  procedures. 
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(A) 
MENTAL  HEALTH  PROGRAMS 

TREATMENT  PROCESS  AND  SERVICES  PROVIDED: 

Sununary  of  Requirements : 

All  of  the  follov/ing  services  are  to  be  provided  unless  the  center 
has  received  written  approval  to  not  provide  the  indicated  service 
from  the  National  Institute  of  Mental  Health  (NIMH)  and  the  Depart- 
ment of  Institutions. 

-Emergency  Services: 

The  CMHC  must  have  24  hour,  7  days  a  week  emergency 
services.   A  treatment  professional  must  be  on  call  at 
all  times  and  available  for  face-to-face  contact  with  a 
person  requiring  services.   24-hour  telephone  services 
should  be  available.  The  phone  number  should  be  published. 

-Outpatient  Services: 

The  CMHC  must  have  outpatient  services  available  at 
appropriate  times  to  meet  the  mental  health  needs  of 
catchment  area  residents. 

-Partial  Hospitalization  Day  Care: 

Partial  hospitalization  must,  at  a  minimum,  be  available 
four  hours  a  day,  five  days  a  week. 

-Inpatient  Services: 

An  adequate  number  of  beds  to  meet  the  inpatient  mental 
health  needs  of  residents  must  be  available  or  designated 
for  priority  use  for  center  patients. 

-Consultation  and  Education  and  Program  Development  (C/E  and 
P/D)  : 

A  formal  plan  and  procedure  including  staff  assignments 
should  be  available  for  C/E  and  P/D  activities.   This 
plan  should  be  monitored  quarterly  to  ensure  accomplish- 
ment. 

-Transitional  Half -Way  House  Services: 

At  least  two  community  based  8-bed  facilities  should  be 
available  in  each  Region  by  7/78.   A  formal  plan  for 
accomplishment  of  this  objective  should  be  available. 
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-Specialized  Services  for  Indians,  Children  and  the  Elderly: 

At  least  one  full-time  treatment  professional  for  each 
target  population  will  be  available  in  each  Region  by 
1/77.   Functions  will  be  to  develop,  implement  and 
coordinate  programs  for  target  groups. 

-Specialized  Services  for  Drug  Abusers  and  Alcoholics: 

At  least  one  full-time  mental  health  professional  f or ^ 
both  target  populations  will  be  available  in  each  Region 
by  1/77.   Functions  will  be  to  develop,  implement  and 
coordinate  programs  for  these  target  groups. 

-Screening  and  Evaluation: 

Programs  must  screen  %  (see  Warm  Springs  State 

Hospital  (WSSH)  contracts)  of  all  patients  admitted  to 
VJSSH  during  (July  1,  1976  and  June  30,  1977). 

-Follow-up  Services: 

Programs  should  maintain  a  formal  follow-up  procedure  for 
discharged  clients  within  regular  intervals  during  the  12 
months  following  discharge. 

-Utilization/Peer  Review: 

There  must  be  a  periodic  evaluation  of  patient  service 
needs  by  other  than  the  responsible  therapist  to  deter- 
mine if  the  provision  of  service  is  clinically  sound  and 
appropriate . 

-A  wide  range  of  supportive  services  must  be  available  to 
center  patients  regardless  of  the  program  to  which  they  are 
assigned.   Services  should  include  at  least:    individual, 
family  and  group  counseling,  occupational  and  recreational 
therapy,  and  chemotherapy. 

-CMHCs  must  coordinate  and  arrange  for  services  needed  by 
clients  which  are  not  available  at  the  center  (Referral 
System) . 

(Provision  of  these  services  should  be  documented  in 
client  records  and  be  part  of  individual  treatment 
plans. ) 
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Evaluation  Procedures: 

1 .    Provision  of  Required  Services: 

To  assess  whether  all  required  services  are  in  operation, 
interview  the  treatment  coordinator  and  prepare  descriptions 
of  services  offered.   For  each  service  program  (Inpatient, 
outpatient,  partial  hospitalization,  etc.)  describe  agency 
procedures  for  intake,  counselor  assignment  and  provision  of 
supportive  services  such  as  Individual  and  group  counseling, 
vocational  rehabilitation,  etc. 

NOTE:     Conduct  these  procedures  for  all  satellite 

operations  as  well  as  the  main  mental  health 
center  program 

.Are  all  required  services  provided? 

.If  not,  document  the  reasons  given.   If  required  services 
are  not  offered,  the  program  should  request  a  formal 
waiver  from  NIMH  and  the  Department  of  Institutions. 

.List  the  addresses  of  each  satellite  operation  and  ser- 
vices provided. 

In  addition,  for  each  required  service  program,  including 
supportive  services,  perform  the  appro|ir1ate  steps  as  follows: 
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a.  Emergency  Services: 

Obtain  and  review  the  schedule  of  staff  assigned  for  24 
hour,  7  day  a  week  coverage  for  emergency  services. 

,Is  a  treatment  professional  on  call  for  face-to-face 
contact  at  all  times? 

.Document  the  emergency  services  telephone  number  and 
document  the  method  by  which  it  is  communicated  to 
the  public. 

b.  Outpatient  Services: 

Determine  whether  the  program  has  made  a  formal  study  to 
determine  whether  outpatient  services  are  needed  outside 
the  normal  9-5  work  day.   Review  study  results  and  sum- 
marize findings.  What  are  the  hours  of  operation  for 
outpatient  services?   How  many  treatment  staff  are  nor- 
mally assigned  to  work  hours  outside  the  9-5  work  day? 

c.  Partial  Hospitalization/Day  Care: 

Obtain  a  copy  of  the  day  care  schedule  of  activities  and 
staff  assignments. 
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.Are  services  available  4  hours  per  day,  5  days  per 
week? 

.If  not,  determine  the  reason  and  whether  the  program 
plans  to  increase  day  care  services. 

d.  Inpatient  Services: 

Document  the  number  of  inpatient  beds  available  at  the 
CMHC  and  satellites. 

.If  none,  obtain  a  copy  of  the  service  agreement(s) 
with  local  hospitals. 

.Does  the  agreement  specify  the  number  of  beds  desig- 
nated for  center  patients? 

.Contact  the  appropriate  hospital  personnel  to  verify 
that  the  number  of  indicated  beds  are  available. 

e.  Transitional  or  Half-Way  House  Services: 

Obtain  and  review  formal  plans  for  implementing  two  8-bed 
half-way  house  facilities  by  7/78. 

.Has  the  Board  of  Directors  participated  in  plan 
development? 
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.Based  on  plan  content,  do  the  steps  and  funding 
sources  appear  reasonable? 

f.    Services  to  Special  Target  Groups: 

Identify  the  staff  person  responsible  for  development, 
implementation  and  coordination  of  service  programs  for 
the  following  groups: 

.Indians 

.Elderly  (65  Years  of  Age  or  Older) 

.Children  (18  Years  of  Age  or  Younger) 

.Drug  Abusers 

.Alcoholics 

Review  each  staff  person's  job  description  to  verify  that 
a  formal  plan  for  program  development  and  staff  assign- 
ment has  been  established. 

g.    Screening  and  Evaluation: 

.Describe  the  internal  program  procedures  for: 

— Receiving  client-screening  referrals  from  the 
court. 

— Conducting  pre-referral  screening  and  diagnosis 
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of  patients  potentially  in  need  of  institutional 
care  including  staff  used. 

— Methods  of  recording  screening  results  including 
forms  used. 

.Prior  to  performing  the  mental  health  center  on-site 
review,  obtain  from  Warm  Springs  State  Hospital  a 
list  of  those  persons  admitted  to  Warm  Springs  State 
Hospital  from  the  center's  region  during  the  past 
six  months. 

.During  the  review,  obtain  a  copy  of  the  center's 
contract  with  Warm  Springs  State  Hospital  and  per- 
form the  following  steps: 

— ^Have  center  staff  determined  which  of  the 

listed  Warm  Springs  State  Hospital  clients  were 
screened  and  evaluated  at  the  center  prior  to 
admission  to  Warm  Springs  State  Hospital. 

1 
— Review  case  records  of  these  persons  and  docu- 
ment evidence  that  screening  and  evaluation 
actually  took  place. 

— Complete  the  "Screening  and  Evaluation  Worksheet", 
Exhibit  IV- 1. 
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.Do  results  indicate  that  the  center  is  in  compliance 
with  contract  provisions?   If  not,  confer  with  the 
financial  management  evaluator  on  the  review  team  to 
calculate  the  amount  of  ineligible  payments  received 
during  the  past  six  months. 

h.    Termination  Procedures  and  Follow-up  Services: 

Describe  the  internal  program  procedures  for  documenting 
client  termination  and  for  conducting  follow-up  after  the 
client  is  discharged  from  the  CMHC .   Obtain  access  to 
client  case  records  of  discharged  or  inactive  clients. 

•To  verify  that  stated  program  procedures  are  actually 
operating,  select  a  sample  of  at  least  15  inactive 
clients  discharged  in  the  last  six  months  and  com- 
plete the  "Termination  and  Follow-up  Documentation 
Worksheet",  Exhibit  IV-2.  (From  the  record,  docu- 
ment the  program  to  which  the  client  was  referred, 
if  any.   See  below.) 

.Do  results  indicate  that  client  terminations  are 
adequately  documented  and  that  the  program  has 
Implemented  a  client  follow-up  system? 

.Does  it  appear  that  the  agency  utilizes  other  ser- 
vice programs  for  client  referral? 
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1.   Utilization  and  Peer  Review: 

Describe  procedures  for  peer  review  of  each  client's 
treatment  plan  and  progress. 

.Are  results  of  peer  review  documented  in  client  case 
records? 

.During  review  of  active  client  case  records  for  the 
most  recent  month  (see  below)  verify  that  peer 
review  procedures  are  actually  in  operation  and  that 
results  are  documented  in  case  records. 

j.    Other  Supportive  Services: 

Describe  all  supportive  services  made  available  to  pro- 
gram clients. 

.Are  dates  of  service  contact  recorded  in  client  case 
records?   How? 

.During  review  of  active  client  case  records  for  the 
most  recent  month,  (see  below)  verify  and  document 
the  frequency  of  supportive  service  contact  for  each 
service  provided. 

k.   Referral  Services: 
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Describe  the  methods  used  for  referring  program  clients 
to  other  service  agencies  including: 

.A  list  of  the  most  common  agencies  used. 

•Forms  used,  if  any.   (Obtain  copy.) 

.Follow-up  procedures  to  determine  results  of  the 
referral. 

.Does  it  appear  that  the  program  actively  utilizes 
other  service  programs  for  their  clients? 

.Verify  that  program  procedures  are  operating  by  data 
collected  on  the  "Termination  and  Follow-up  Docu- 
mentation Worksheet"  Exhibit  IV-2.   (See  above.) 

1.    Community  Consultation  and  Education  Activities: 

To  assess  the  extent  to  which  the  agency  plans  and  moni- 
tors community  consultation  and  education  activities, 
interview  the  program  director  and  perform  the  following 
steps: 

a.   Describe  those  community  consultation  and  education 
(C/E)  activities  conducted  during  the  past  six 
months.   Include  in  the  description  the  following: 
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-The  purpose  or  objectives  of  the  C/E  activities. 
-The  names  of  community  groups  or  agencies 

contacted. 
-The  number  and  type  of  program  staff  conducting 

C/E  activities. 

b.  Obtain  the  most  recent  monthly  report  of  staff  hours 
spent  on  C/E  activities.   Trace  the  hours  reported 
back  to  detailed  supporting  documentation  prepared 
by  staff  such  as  daily  or  weekly  time  cards  or  logs. 
Are  reported  hours  supported  by  detailed  documenta- 
tion? 

c.  Does  it  appear  that  the  C/E  activities  are  well 
planned  and  monitored?   Consider  the  following: 

-Can  the  agency  show  evidence  that  objectives  of 
C/E  activities  have  been  achieved? 

-Are  progress  reports  prepared  by  staff  con- 
ducting C/E  activities?  Are  reports  submitted 
to  the  program  director? 

-Does  the  frequency  and  nature  of  C/E  activities 
justify  the  number  of  staff  positions  and  amount 
of  time  spent  on  C/E  activities?   Document  the 
rationale  for  your  assessment. 
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B.    CLIENT  CENSUS  AND  REPORTING: 

Summary  of  Requirements: 

Program  reporting  systems  should  produce  accurate  reports  of: 

-The  number  of  active  clients. 

-Other  data  that  may  be  requested  by  the  Department  of  Insti- 
tutions. 

These  reports  should  be  supported  by  adequate  documentation  to 
validate  report  accuracy. 

Evaluation  Procedures: 


1 .    Process  for  Report  Preparation: 

To  assess  the  process  for  report  preparation  and  methods  of 
documentation,  interview  the  individual(s)  responsible  for 
preparation  of  program  reports  (not  financial)  and  perform  the 
following  steps: 

a.  Prepare  a  brief  flow  chart  of  the  steps  used  for  prepara- 
tion of  the  mental  health  center's  monthly  program  report. 
This  chart  should  begin  with  treatment  staff  documenta- 
tion of  clients  served,  services  provided  and  staff  hours 
spent,  and  end  with  submittal  of  the  monthly  report  to 
the  Department  of  Institutions. 

b.  Describe  methods  used,  if  any,  by  program  management  to 
validate  information  supplied  by  treatment  staff. 

NOTE:    If  no  verification  is  conducted,  this  is  a 
weakness  and  should  be  documented. 
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c.  Collect  copies  of  the  mental  health  center's  monthly 
program  reports  for  the  last  six  months.  During  the 
interview  with  the  individual  responsible  for  report 
preparation,  determine  the  following: 

.What  criteria  are  used  for  client  admission  and 
discharge  for  each  of  the  following  programs: 

— Inpatient 

— Outpatient 

— Residential 

— Partial  Hospitalization  (Day  Care) 

.What  criteria  are  used  for  determination  of  active 
versus  inactive  clients  regarding  frequency  of 
contact? 

d.  Assess  whether  the  system  used  is  reasonable.   Interview 
3-5  professional  staff  to  verify  that  reporting  procedures 
and  definitions  as  described  above  have  been  communicated 
to  staff.   (See  the  "Professional  Staff  Interview  Question- 
naire", Exhibit  II-2,  in  the  Personnel  Management  and 
Staff  Development  Review  Guide.) 

2 .    Accuracy  of  Reports: 

To  assess  the  accuracy  of  the  mental  health  center's  monthly 
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report  of  active  outpatients  and  day-care  clients,  document 
of  the  number  of  clients  actually  served  in  those  programs 
for  the  most  recent  month. 

NOTE :  The  client  census  review  should  be  conducted 
simultaneously  with  the  client  record  reviev/ 
as  described  in  the  next  section. 

a.    Using  the  individual  client  case  record  or  other  qualified 
source  (medication  dispensing  logs,  group  therapy  atten- 
dance logs,  etc.)  complete  the  "Client  Census  Documenta- 
tion Worksheet",  Exhibit  IV-3. 

.To  obtain  the  list  of  active  outpatients  and  day- 
care clients  the  evaluator  should  request  either  a 
centralized  list  of  active  clients  or  access  to  the 
active  client  files.   The  evaluator  should  include 
as  active  those  clients  discharged  during  the  cur- 
rent month  who  were  active  during  the  test  month. 
The  evaluator  should  find  records  available  for  at 
least  the  number  of  outpatients  and  day-care  clients 
reported  on  the  monthly  report . 

.Enter  the  client  case  number  for  each  reported 
active  outpatient  and  day-care  client  on  the  "Client 
Census  Documentation  Worksheet",  Exhibit  IV-3. 
Review  documentation  supplied  by  the  program  and 
enter  a  check  mark  (  )  in  the  column  next  to  the 
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EXHIBIT  IV- 3 


CLIENT  CENSUS  DOCUMENTATION  WORKSHEET 


Program  Name : 
Reviewed  BY: 


Service  Program: 


CLIENT  I.D. 

MOST  RECENT  MONTH 
(  ) 

SECOND  TEST  MONTH 

THIRD  TEST  MONTH 

— 

TOTAL  THIS  PAGE 
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client  number  FOR  EACH  CLIENT  FOR  WHOM  AT  LEAST  ONE 
FACE-TO-FACE  CONTACT  IS  DOCUMENTED  FOR  THE  TEST 
MONTH (S) . 

Compare  the  documented  monthly  census  by  service  program 
(outpatient  or  day-care)  to  the  reported  census  for  the 
same  month. 

.Complete  the  "Documented  and  Reported  Census  Recon- 
ciliation Worksheet",  Exhibit  IV-4,  for  outpatient 
and  day-care  programs. 

.For  any  site  or  service  program  with  a  reported 
versus  documented  census  variance  of  more  than  10%, 
review  results  of  analysis  and  probable  causes  of 
the  problem  with  program  management.   Some  possible 
causes  of  the  problem  might  be: 

— Failure  of  treatment  staff  to  terminate  clients 
on  a  timely  basis. 

— Poor  procedures  for  documenting  client  contacts 
in  case  records  or  other  sources. 

.Complete  an  additional  two  months  of  census  docu- 
mentation for  programs  and  sites  with  variances  of 
greater  than  10%. 
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c.    Calculate  the  average  client  census  as  follows: 

.For  those  programs  and  sites  with  less  than  10% 
variance  in  reported  versus  documented  census, 
calculate  the  average  census  for  the  past  six  months 
using  mental  health  center  monthly  reports. 

.For  those  programs  and  sites  with  more  than  a  10% 
variance,  calculate  the  average  census  for  the  three 
months  reviewed. 

3.    Average  Utilization  of  Residential  and  Inpatient  Beds: 

To  calculate  the  average  utilization  of  residential  and 
inpatient  facilities  perform  the  following  steps: 

a.  Obtain  attendance  records  from  residential  or  inpatient 
facilities  and  determine: 

.Number  of  beds  available  during  the  month. 
.Number  of  bed-days  used  during  the  month. 

b.  Derive  average  utilization  for  the  month  by  dividing  the 
number  of  bed-days  used  during  the  month  by  the  number  of 
bed  days  available  during  the  month.   (Number  of  bed-days 
available  during  the  month  =  number  of  beds  available 
during  the  month  x  number  of  days  in  the  month.) 
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c.    If  the  average  utilization  is  relatively  low  (70%  or 
less)  develop  an  assessment  of  why  the  facilities  are 
underutilized . 

(Consider) 

.Does  the  client  turnover  rate  appear  to  be  excessive? 

CLIENT  RECORDS: 

Summary  of  Requirements: 

Active  client  records  should  contain: 

-Individualized  client  treatment  plans  and  progress  notes. 

-Drug  use  profile.   (Illicit  Drugs) 

-Documentation  of  all  face-to-face  service  contacts. 

-Past  and  present  health  status. 

-A  record  of  client  admission  date. 

In  addition,  client  records  should  be  well  organized,  uniformly 
maintained,  and  stored  in  locked  file  cabinets  in  compliance  with 
confidentiality  requirements. 

Evaluation  Procedures : 

The  review  of  client  records  is  to  accomplish  three  primary  ob- 
jectives as  follows: 

.To  document  the  frequency  and  duration  of  client  treatment 
contact. 

.To  determine  whether  case  records  contain  required  client 
treatment  documentation. 
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.To  assess  the  quality  of  case  record  organization  and  mainte- 
nance including  record  storage. 

To  meet  these  objectives,  the  evaluator  should  perform  the  following 
evaluation  procedures: 

1 .    Methods  of  Recording: 

To  understand  the  program's  record  keeping  system,  interview 
the  program  treatment  coordinator  and  perform  the  following 
steps: 

a.  Obtain  2-3  sample  active  client  records  and  review  each 
record  with  the  treatment  coordinator.   Describe  the 
forms  used  for  the  following: 

.Client  Demographic  Data 

.Drug  Use  Profile  (Illicit  Drugs) 

.Client  Health  History  (physical  and  mental) 

.Problem  Assessment  and  Admission  Date 

.Individualized  Treatment  Plan 

.Plan  Assessment  and  Progress  Notes 

.Results  of  Peer  Review 

.Documentation  of  all  Supportive  Services  Provided 

•Other 

b.  Obtain  a  copy  of  each  form. 
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2.    Record  Content: 

To  evaluate  the  content  of  client  records,  obtain  access  to 
files  of  active  client  case  records  and  perform  the  following 
steps : 

a.  Select  a  sample  of  case  records  for  presently  active 
clients  not  to  exceed  100  records.   (If  the  program  has 
fewer  than  100  active  clients,  document  record  content 
for  all  clients.) 

NOTE:     For  programs  with  more  than  100  clients, 

randomly  select  client  records  for  review. 
For  example,  for  programs  with  500  or  more 
clients,  select  every  fifth  record,  etc. 

b.  Utilizing  the  "Client  Record  Documentation  VJorksheet", 
Exhibit  IV-5,  document  record  content  for  each  program 
type  and  site  as  follows: 

.Client  Number: 

The  evaluator  is  cautioned  to  ensure  client 
confidentiality  by  utilizing  only  the  client 
identification  number.   Names  may  be  used  on- 
site  but  should  be  erased  from  worksheets 
before  leaving. 
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.Admission  Date: 

To  calculate  average  client  retention  (see 
below)  enter  the  most  recent  admission  date  for 
each  client. 

.Content  of  Records: 

Based  on  the  description  of  program  procedures, 
look  for  the  presence  in  each  file  of  the 
indicated  documentation. 

.Frequency  of  Supportive  Services  Contact: 

Record  the  number  of  documented  face-to-face 
contacts  by  type  of  service  for  the  most  recent 
month.   Documentation  should  consist  of  written 
counseling  notes  or  other  qualified  sources 
such  as  medication  dispensing  logs  or  group 
counseling  attendance  records.   The  evaluator 
should  request  the  assistance  of  program  staff 
to  locate  all  reasonably  accessible  documenta- 
tion of  contact.   Family  counseling  contacts 
need  not  include  the  primary  client  as  a  parti- 
cipant. 

NOTE:     To  help  insure  that  clients  are 
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receiving  necessary  services  and 
that  staff  time  is  efficiently 
utilized,  all  supportive  service 
contacts  should  be  documented  in 
case  records  as  they  occur, 
e.g.,  case  files  for  day-care 
clients  should  include  documenta- 
tion of  individual  service 
contacts ,  not  just  documentation 
of  service  days  received.   If 
individual  contacts  are  not 
documented,  this  is  a  weakness, 
and  should  be  noted. 

c.    To  assess  the  extent  to  which  progress  notes  in  client 
records  are  consistent  with  individualized  treatment 
plans  and  the  extent  to  which  the  updating  of  these 
treatment  plans  reflects  progress  which  has  been  re- 
ported, perform  the  following  steps: 

•Select  a  sample  of  fifteen  (15)  client  case  records 
from  the  "Client  Record  Documentation  Worksheet", 
Exhibit  IV-5,  and  perform  the  following  steps: 

.Review  the  individualized  treatment  plan  and  pro- 
gress notes  for  each  case  record.   Determine  whether 
they  are  consistent  with  one  another,  i.e.,  whether 
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progress  toward  the  specific  objectives  in  the 
treatment  plan  is  mentioned  in  the  notes,  and  whether 
treatment  staff  recommendations  become  part  of  the 
plan  objectives  which  are  mentioned  in  subsequent 
progress  notes. 

.Are  progress  notes  consistent  with  treatment  plans? 

d.    Summarize  results  of  the  record  documentation  review  by 
calculating  the  following  statistics: 

.Client  Retention: 

Based  on  the  date  of  admission,  determine  the 
number  and  percent  of  clients  who  have  been 
active  for: 

0-1  Month  (Less  than  30  Days) 

1-3  Months  (30-90  Days) 

3-6  Months 

6-+  Months  I 

.Record  Content: 

Calculate  the  number  and  percent  of  client 
records  lacking  the  required  documentation. 
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.Peer  Review: 

For  those  client<3  active  more  than  three  months, 
calculate  the  number  and  percent  of  client 
records  indicating  that  "peer  review"  has  not 
been  conducted. 

.Frequency  of  Supportive  Services  Contact  by  Client: 

Calculate  the  number  and  percent  of  clients 
provided  services  the  following  number  of  times 
during  the  most  recent  month: 


0 

1-3 

4-6 

7-9 

10-+ 


.Frequency  of  Supportive  Services  Contact  by  Admis- 
sion Date  of  Client: 

Calculate  the  number  and  percent  of  clients  in 
each  of  the  retention  categories  provided 
services  in  each  of  the  contact  frequency 
categories.   Complete  the  "Frequency  of  Sup- 
portive Services  Contact  by  Admission  Date  of 
Client  Worksheet",  Exhibit  IV-6. 
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.Frequency  of  Supportive  Services  Contact  by  Service: 

Calculate  the  total  number  and  percent  of 
contacts  provided  to  all  clients  by  service 
during  the  most  recent  month  as  follows : 

TYPE  OF  SERVICE     NO.  OF  CONTACTS      %  OF  TOTAL  CONTACTS 

Individual 
Counseling 

Group  Counseling 

Non-Intake  Medical 
Services 

Vocational  Rehab- 
ilitation, etc. 

TOTAL  100% 

e.    Analyze  results  of  the  record  documentation  review. 
Consider  the  following: 

.How  does  the  program  compare  with  other  programs  in 
client  retention? 

.Is  required  record  content  present  for  at  least  90% 
of  all  records  reviewed?   If  not,  discuss  reasons 
with  program  management. 

.Does  it  appear  that  "peer  review"  was  conducted  for 
at  least  75%  of  clients? 
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.Do  records  accurately  reflect  the  extent  and  nature 
of  supportive  services  provided?   How  does  the 
program  compare  with  other  programs  in  frequency  of 
contact? 

•Does  it  appear  that  newer  clients  have  a  greater 
frequency  of  service  contact? 

.Are  overall  service  contacts  by  type  reasonably 
distributed  or  does  it  appear  that  only  one  or  two 
types  of  services  are  being  provided  and  documented? 

.Are  records  stored  in  locked  file  cabinets?   Do 
records  appear  well  organized  and  uniformly  main- 
tained? 

.Review  all  findings  with  program  management  and 
identify  reasons  for  problems  observed  and  solutions 
program  management  intends  to  implement. 

NOTE:    These  solutions  should  become  part  of 
the  program's  treatment  objectives  for 
the  next  program  year. 
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D.    PATIENT  ELIGIBILITY ; 


S Ultima ry  of  Requirements : 


Those  patients  released  from  V7SSH  for  whom  CriHCs  are  claiming 
reimbursement  for  services  provided  under  Article  III  -  NSSH 
Contract,  must  meet  the  following  patient  eligibility  criteria: 

-They  must  have  been  residents  in  WSSH  on  6/30/75. 

-A  individualized  treatment  plan  must  be  developed  by  WSSH  and 
crffiC  staff.  (Only  required  for  patients  released  on  or  after 
May  13,  1976.) 

-Approval  for  services  must  be  obtained  by  the  Department  of 
Institutions,  Mental  Health  Bureau. 

Evaluation  Procedures: 


To  assess  the  mental  health  center's  compliance  with  contract 
provisions  for  patient  eligibility,  perform  the  following  steps: 

.Obtain  from  the  Mental  Health  Bureau  or  from  the  CMHC,  if 
necessary,  a  copy  of  the  most  recent  month's  statement/claim 
for  reimbursement  under  Article  III  of  the  Warm  Springs  State 
Hospital  contract.   This  statement  should  include  both  a  list 
of  the  number  of  clients  and  services  by  type,  which  are 
claimed,  and  a  costing  of  these  services. 

.Obtain  from  the  CMHC  supporting  documentation  for  the  claim, 
including  a  list  of  all  clients  served. 

.Review  the  case  records  of  all  clients  claimed  under  Article 
III  for  documented  evidence  of  the  above  patient  eligibility 
criteria. 
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NOTE:  A  copy  of  the  eligibility  checklist  prepared  by 
Warm  Springs  State  Hospital  should  be  contained 
in  each  file. 

Complete  the  "Client  Eligibility  Worksheet",  Exhibit  IV-7  and 
determine  the  percentage  of  claimed  clients  who  were  ineligible, 

.For  each  ineligible  client,  review  documentation  of  all  ser- 
vices received  to  determine  the  specific  service  charges  to  be 
disallowed.   Sum  these  charges  over  all  ineligible  clients  to 
determine  the  total  amount  of  charges  to  be  disallowed  for  the 
center. 
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(B) 
DRUG  ABUSE  TREATMENT  PROGRAMS 

TREATMENT  PROCESS  AND  SERVICES  PROVIDED: 
Summary  of  Requirements: 

Supportive  services  which  must  be  provided  for  outpatient  and 
residential  programs  are: 

-Educational 

-Vocational  Counseling  and  Training 

-Job  Development  and  Placement 

-Legal  Services 

-Referral  Services 

-Follow-up  Services 

-Individual,  Family  and  Group  Counseling 

-Medical  Services 

Evaluation  Procedures: 


To  assess  whether  all  required  supportive  services  are  available, 
interview  the  treatment  coordinator  and  prepare  descriptions  of  the 
services  offered. 

.Are  all  required  services  available? 

.During  review  of  active  client  case  records  for  the  most 
recent  month  (see  below)  verify  and  document  the  frequency  of 
supportive  service  contact  for  each  service  provided. 

In  addition,  perform  the  following  steps  for  vocational  and  educa- 
tion services  and  follow-up  and  referral  services. 
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1 .  Vocational  and  Education  Services: 

To  assess  the  extent  to  which  vocational  and  education  ser- 
vices have  resulted  In  client  placement,  perform  the  following 
steps: 

a.  Complete  the  "Job  and  Education  Placement  Analysis  Work- 
sheet", Exhibit  IV-8. 

.Obtain  a  list  of  clients  placed  in  jobs  or  that  have 
achieved  an  Increased  level  of  education  during  the 
past  six  months. 

.Document  job  placement  or  educational  achievement 
from  client  case  records  or  another  qualified  data 
source. 

2.  Termination  Procedures  and  Follow-up  and  Referral  Services: 

To  assess  the  extent  to  which  a  system  of  client  termination 
and  follow-up  and  a  system  of  referring  clients  to  other 
service  agencies  have  been  Implemented,  perform  the  following 
steps: 

a.  Describe  the  internal  program  procedures  for  documenting 
client  termination  and  conducting  client  follow-up  after 
discharge  from  the  treatment  facility.  Obtain  access  to 
case  records  of  discharged  or  inactive  clients. 
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EXHIBIT  IV-8 


JOB  AND  EDUCATION  PLACEMENT  ANALYSIS  WORKSHEET 


Program  Name: 
Reviewed  By: 
Date: 


For  Clients  Active  in  the  Month  of: 


CLIENT 
I.D. 

TYPE  OF  JOB  OR 
EDUCATIONAL  LEVEL  ACHIEVED 

DATE  PLACED 
OR  ACHIEVED 

PLACED  BY 

TOTAL 
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.To  verify  that  stated  program  procedures  are  actually 
operating,  select  a  sample  of  15  inactive  clients 
discharged  in  the  last  six  months  and  complete  the 
"Termination  and  Follow-up  Documentation  Worksheet", 
Exhibit  IV-9. 

.Do  results  indicate  that  the  program  has  implemented 
a  client  follow-up  system? 

.Are  reasons  for  client  termination  documented? 

b.    Describe  the  methods  used  for  referring  program  clients 
to  other  service  agencies  including: 

.A  list  of  the  most  common  agencies  used. 

.Forms  used,  if  any.   (Obtain  copy.) 

•Follow-up  procedures  to  determine  results  of  the 
referral. 

.Does  it  appear  that  the  program  actively  utilizes 
other  service  programs  for  their  clients? 

.Verify  that  program  procedures  are  operating  by  data 
collected  on  the  "Termination  and  Follow-up  Documenta- 
tion Worksheet",  Exhibit  IV-9.   (See  above.) 
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CLIENT  CENSUS  AND  REPORTING 

Summary  of  Requirements: 

Local  CODAP  reporting  systems  should  produce  accurate  reports  of: 

-The  number  of  active  clients  at  the  end  of  each  month  defined 
as  those  receiving  face-to-face  contacts  within  the  past  30 
days. 

-Other  data  that  may  be  requested  by  the  Department  of  Insti- 
tutions. 

These  reports  should  be  supported  by  adequate  documentation  to 
validate  report  accuracy. 

Evaluation  Procedures: 


1 .    Process  for  Report  Preparation: 

To  assess  the  process  for  report  preparation  and  methods  of 
documentation,  interview  the  individual (s)  responsible  for 
preparation  of  program  reports  (not  financial)  and  perform  the 
following  steps: 

a.  Prepare  a  brief  flow  chart  of  the  steps  used  for  prepara- 
tion of  the  CODAP  Client  Flow  Summary.   This  chart  should 
begin  with  treatment  staff  documentation  of  clients 
served,  services  provided  and  staff  hours  spent,  and  end 
with  submittal  of  the  monthly  report  to  the  Department  of 
Institutions. 

b.  Describe  methods  used.  If  any,  by  program  management  to 
validate  information  supplied  by  treatment  staff. 
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NOTE:    If  no  verification  is  conducted,  this  is 
a  weakness  and  should  be  documented. 

c.  Collect  copies  of  the  CODAP  Client  Flow  Summary  Report 
for  the  last  six  months.   During  the  interview  with  the 
individual  responsible  for  report  preparation,  determine 
the  following: 

.What  criteria  are  used  for  client  admission  and  dis- 
charge for  each  of  the  following  programs: 

— Inpatient 

— Outpatient  (Drug  Free) 

— Residential 

— Outpatient  Detoxification 

d.  Assess  whether  the  system  used  is  reasonable.   Interview 
3-5  professional  staff  to  verify  that  reporting  procedures 
and  definitions  as  described  above  have  been  communicated 
to  staff.   (See  the  "Professional  Staff  Interview  Ques- 
tionnaire", Exhibit  II-2,  in  the  Personnel  Management  and 
Staff  Development  Review  Guide.) 

2  .   Accuracy  of  Reports: 

To  assess  the  accuracy  of  the  most  recent  CODAP  Client  Flow 
Summary  Report,  documentation  of  the  number  of  outpatients 
served  for  the  most  recent  month  should  be  performed  as  follows: 
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NOTE:  The  client  census  review  should  be  conducted 
simultaneously  with  the  client  record  review 
as  described  in  the  next  section. 

Using  the  individual  client  case  record  or  other  qualified 
source  (counselor's  daily  log,  group  therapy  attendance 
logs,  etc.)  complete  the  "Client  Census  Documentation 
Worksheet",  Exhibit  IV-10. 

.To  obtain  the  list  of  active  outpatients  the  evaluator 
should  request  either  a  centralized  list  of  active 
clients  or  access  to  the  active  client  files.   The 
evaluator  should  include  as  active  those  clients 
discharged  during  the  current  month  who  were  active 
during  the  test  month.   The  evaluator  should  find 
records  available  for  at  least  the  number  of  out- 
patients reported  on  the  monthly  CODAP  report  for 
each  treatment  environment. 

.Enter  the  client  case  number  for  each  reported 
active  outpatient  on  the  "Client  Census  Documenta- 
tion Worksheet",  Exhibit  IV-10.   Review  documenta- 
tion supplied  by  the  program  and  enter  a  check  mark 
(yO  in  the  column  next  to  the  client  number  FOR  EACH 
CLIENT  FOR  WHOM  AT  LEAST  ONE  FACE-TO-FACE  CONTACT  IS 
DOCUMENTED  FOR  THE  TEST  MONTH (S) . 
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EXHIBIT   IV- 10 


CLIENT  CENSUS   DOCLF.-tSNTATION  WORKSHEET 


Program  Name; 
Reviewed  BY: 


Service   Program: 


(Outpatient   Drug-Free   or  Outpatient   Detoxification) 


CLIENT  I.D. 

MOST  RECENT  MONTH 
(  ) 

SECOND  TEST  ^«)NTH 

THIRD  TEST  >DNTH 

TOTAL  TKIS  PAGE 

1 
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b .    Compare  the  documented  monthly  census  by  enviroment  to 
the  reported  census  for  the  same  month. 

.Complete  the  "Documented  and  Reported  Census  Recon- 
ciliation Worksheet",  Exhibit  IV-U. 

•For  any  site  or  environment  with  a  reported  versus 
documented  census  variance  of  more  than  10%,  review 
results  of  analysis  and  probable  causes  of  the 
problem  with  program  management.   Some  possible 
causes  of  the  problem  might  be: 

— Failure  of  treatment  staff  to  terminate  clients 
on  a  timely  basis. 

— Poor  procedures  for  documenting  client  contacts 
in  case  records  or  other  sources. 

.Complete  an  additional  one  month  of  census  docu- 
mentation for  programs  and  sites  with  variances  of 
greater  than  10%. 

c.   Calculate  the  average  client  census  as  follows: 

.For  those  programs  and  sites  with  less  than  10% 
variance  in  reported  versus  documented  census,  cal- 
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culate  the  average  census  for  the  past  six  months 
using  CODAP  Client  Flow  Summary  Report. 

•For  those  programs  and  sites  with  more  than  a  10% 
variance,  calculate  the  average  census  for  the  two 
months  reviewed. 

3.    Average  Utilization  of  Residential  and  Inpatient  Beds: 

To  calculate  the  average  utilization  for  residential  and 
inpatient  facilities  perform  the  following  steps: 

a.  Obtain  attendance  records  from  residential  or  inpatient 
facilities  and  determine: 

.Number  of  beds  available  during  the  month. 
.Number  of  bed  days  used  during  the  month. 

b.  Derive  average  utilization  for  the  month  by  dividing  the 
number  of  bed  days  used  during  the  month  by  the  number  of 
bed  days  available  during  the  month.   (Number  of  bed  days 
available  during  the  month  =  number  of  beds  available 
during  the  month  x  number  of  days  in  the  month.) 

c.  If  the  average  utilization  is  relatively  low  (70%  or 
less)  develop  an  assessment  of  why  the  facilities  are 
underutilized . 
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(Consider) 


•Does  the  client  turnover  rate  appear  to  be  exces- 
sive? 


CLIENT  RECORDS: 

Summary  of  Requirements : 

All  client  records  must  include  the  following  required  documenta- 
tion: 

-Individualized  Treatment  Plan 

-Results  of  a  90  day  plan  assessment  for  outpatients  and  a  30 

day  plan  assessment  for  residents. 
-Monthly  progress  reports  and  documentation  of  all  supportive 

service  contacts. 
-Physical  and  Laboratory  Examinations 
-Personal  History 
-Medical  History 
-Drug  History 
-Results  of  Urine  Testing. 

In  addition,  client  records  should  be  well  organized,  uni- 
formly maintained,  and  stored  in  locked  file  cabinets  in 
accordance  with  confidentiality  requirements. 

Evaluation  Procedures: 


The  review  of  client  records  is  to  accomplish  three  primary  objec- 
tives as  follows: 

.To  document  the  frequency  and  duration  of  client  treatment 
contact. 

.To  determine  whether  case  records  contain  required  client 
treatment  documentation. 
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•To  assess  the  quality  of  case  record  organization  and  main- 
tenance Including  record  storage. 

To  meet  these  objectives,  the  evaluator  should  perform  the  fol- 
lowing evaluation  procedures: 

1 .   Methods  of  Recording: 

To  understand  the  program's  record  keeping  system,  interview 
the  program  treatment  coordinator  and  perform  the  following 
steps: 

a.  Obtain  2-3  sample  active  client  records  and  review  each 
record  with  the  treatment  coordinator.   Describe  the 
forms  used  for  the  following: 

.Client  Demographic  Data 
.Personal  History 
.Drug  History 
•Medical  History 

.Physical  and  Laboratory  Examinations,  including 
results  of  Urine  Testing 
.Individualized  Treatment  Plan 
.Plan  Assessment  and  Progress  Reports 
.Documentation  of  Services  Provided 
.Other 

b.  Obtain  a  copy  of  each  form. 
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2.    Record  Content: 

To  evaluate  content  of  client  records,  obtain  access  to  files 
of  active  client  case  records  and  perform  the  following  steps: 

a.  Select  a  sample  of  case  records  for  presently  active 
clients  not  to  exceed  100  records.   (If  the  program  has 
fewer  than  100  active  clients,  document  record  content 
for  all  clients.) 

nOTE :     For  programs  with  more  than  100  clients, 

randomly  select  client  records  for  review. 
For  example,  for  programs  with  500  or  more 
clients,  select  every  fifth  record,  etc. 

b.  Utilizing  the  "Client  Record  Documentation  Worksheet", 
Exhibit  IV-12,  document  record  content  for  each  program 
type  and  site  as  follows: 

•Client  Number: 

The  evaluator  is  cautioned  to  ensure  client 
confidentiality  by  utilizing  only  the  client 
identification  number.   Names  may  be  used  on- 
site  but  should  be  erased  from  worksheets 
before  leaving. 
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.Admission  Date: 

To  calculate  average  client  retention  (see 
below)  enter  the  admission  date  for  each 
client. 

.Content  of  Records: 

Based  on  the  description  of  program  procedures, 
look  for  the  presence  in  each  file  of  the 
indicated  documentation. 

.Frequency  of  Supportive  Services  Contact: 

Record  the  number  of  documented  face-to-face 
contacts  by  type  of  service  for  the  most  recent 
month.   Documentation  should  consist  of  written 
counseling  notes  or  other  qualified  sources 
such  as  group  counseling  attendance  records. 
The  evaluator  should  request  the  assistance  of 
program  staff  to  locate  all  reasonably  acces- 
sible documentation  of  contact.   Family  coun- 
seling contacts  need  not  include  the  primary 
client  as  a  participant. 

MOTE:     To  help  ensure  that  clients  are 
receiving  necessary  services  and 


I 
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that  staff  time  is  efficiently 
utilized,  all  supportive  service 
contacts  should  be  documented  in 
case  records  as  they  occur, 
e.g.,  case  files  for  residential 
clients  should  include  documenta- 
tion of  individual  service 
contacts,  not  just  documentation 
of  service  days  received.   If 
individual  contacts  are  not 
documented,  this  is  a  weakness 
and  should  be  noted. 

To  assess  the  extent  to  which  progress  notes  in  client 
records  are  consistent  with  individualized  treatment 
plans  and  the  extent  to  which  the  updating  of  these 
treatment  plans  reflects  progress  which  has  been  re- 
ported, perform  the  following  steps: 

.Select  a  sample  of  15  client  case  records  from  the 
"Client  Record  Documentation  Worksheet",  Exhibit  IV- 
12,  and  perform  the  following  steps: 

.Review  the  individualized  treatment  plan  and  progress 
notes  for  each  case  record.   Determine  whether  they 
are  consistent  with  one  another,  i.e.,  whether 
progress  toward  the  specific  objectives  in  the 
treatment  plan  is  mentioned  in  the  notes,  and  whether 
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treatment  staff  recommendations  become  part  of  the 
plan  objectives  which  are  mentioned  in  subsequent 
progress  notes. 

.Are  progress  notes  consistent  with  treatment  plan? 

•Has  the  treatment  plan  been  assessed  within  90  days 
for  outpatients  and  within  30  days  for  residents? 

.Have  there  been  at  least  monthly  progress  notes? 

d.    Summarize  results  of  the  record  documentation  review  by 
calculating  the  following  statistics: 

.Client  Retention: 

Based  on  the  date  of  admission,  determine  the 
number  and  percent  of  clients  who  have  been 
active  for: 

0-1  Month  (Less  than  30  days) 
1-3  Months  (30-90  days) 
3-6  Months 
6-+  Months 

.Record  Content: 
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Calculate  the  number  and  percent  of  client 
records  lacking  the  required  documentation. 

.Frequency  of  Supportive  Services  Contact  by  Client: 

Calculate  the  number  and  percent  of  clients 
provided  services  the  following  number  of  times 
during  the  most  recent  month: 


0 
1-3 

4-6 
7-9 

10-+ 


.Frequency  of  Supportive  Services  Contact  by  Admis- 
sion Date  of  Client: 

Calculate  the  number  and  percent  of  clients  in 
each  of  the  retention  categories  provided 
services  in  each  of  the  contact  frequency 
categories.   Complete  the  "Frequency  of  Sup- 
portive Services  Contact  by  Admission  Date  of 
Client  Worksheet",  Exhibit  IV- 13. 

.Frequency  of  Supportive  Services  Contact  by  Service: 
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Calculate  the  total  number  and  percent  of 
contacts  provided  to  all  clients  by  service 
during  the  mopt  recent  month  as  follows: 

TYPE  OF  SERVICE     NO.  OF  CONTACTS      %  OF  TOTAL  CONTACTS 

Individual 
Counseling 

Group  Counseling 

Family  Counseling 

Vocational 
Education 

Non-Intake  Medical 
Services,  Etc. 

TOTAL 

e.   Analyze  results  of  the  record  documentation  review. 
Consider  the  following: 

.How  does  the  program  compare  with  other  programs  in 
client  retention? 

.Is  required  content  present  for  at  least  90%  of  all 
records  reviewed?   If  not,  discuss  reasons  with 
program  management . 

.Do  records  accurately  reflect  the  extent  and  nature 
of  supportive  services  provided?   How  does  the 
program  compare  with  other  programs  in  frequency  of 
contact? 
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.Does  it  appear  that  newer  clients  have  a  greater 
frequency  of  service  contact? 

.Are  overall  service  contacts  by  type  reasonably 
distributed  or  does  it  appear  that  only  one  or  two 
types  of  services  are  being  provided  and  documented? 

.Are  records  stored  in  locked  file  cabinets?   Do 
records  appear  well  organized  and  uniformly  main- 
tained? 

.Review  all  findings  with  program  management  and 
identify  reasons  for  problems  observed  and  solutions 
program  management  intends  to  Implement. 

NOTE:     These  solutions  should  become  part 

of  the  program's  treatment  objectives 
for  the  next  program  year. 


URINE  SURVEILLANCE 

Summary  of  Requirements: 

For  residential  programs,  urine  samples  must  be  taken  weekly. 
Urine  specimens  must  be  analyzed  at  least  monthly  for  opiates, 
methadone,  amphetamines  and  barbiturates,  in  laboratories  complying 
with  all  applicable  federal  proficiency  testing  standards. 

Evaluation  Procedures: 


1.   Urine  Surveillance  Procedures: 
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To  obtain  a  description  of  urine  testing  procedures,  interview 
the  Program  Treatment  Coordinator  and  determine  the  following: 

a.  Who  is  required  to  provide  urine  samples? 

b.  For  each  service  (inpatient,  residential)  determine  the 
frequency  with  which  clients  must  provide  samples. 

c.  Which  laboratory  is  used  for  urine  screening?   At  what 
cost? 

d.  Do  procedures  comply  with  the  above  stated  rules? 

2 .    Verification  of  Urine  Surveillance  Procedures; 

To  verify  procedures  and  to  assess  results  of  urinalysis, 
perform  the  following  steps: 

a.  Complete  the  "Urinalysis  Documentation  Worksheet", 
Exhibit  IV-14  for  clients  active  in  the  most  recent 
month.   (Use  individual  test  reports  or  summary  logs.) 

b.  Calculate  the  following  statistics: 

.Positive  tests  as  a  percentage  of  all  tests  given  in 
the  most  recent  month. 
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.Clients  with  positive  tests  as  a  percentage  of  all 
tested  clients. 

.Clients  with  positive  tests  as  a  percentage  of  all 
active  clients. 

c.  Determine  the  participation  of  the  urinalysis  laboratory 
in  the  Center  for  Disease  Control  (CDC)  testing  program. 

.Telephone  the  laboratory  and  obtain  the  most  recent 
proficiency  rating  and  document  results. 

d.  Do  results  indicate  that  urine  screening  procedures  are 
in  operation? 

e.  How  do  urinalysis  statistics  compare  with  other  programs? 
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EXHIBIT  IV- 14 

URINALYSIS  DOCUMENTATION  WORKSHEET 
(MONTH) 


Page 


of 


Program  Name: 
Reviewed  BY:_ 
Date: 


Cost  Per  Test:_ 

Testing  Facility:^ 

Types  of  Drugs  Tested  for; 


CLIENT 
NUMBER 


1. 
2. 
3. 
4. 
5. 
6. 


NEGATIVE 


NUMBER  OF  URINALYSIS  TESTS 
POSITIVE 


OPIATES 


OTHER  DRUGS 


TOTAL  NUI-IBER  OF 
URINALYSIS  TESTS 
NEGATIVE  AND  POSITIVE 


L 
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(C) 
ALCOHOLISM  TREATMENT  PROGRAMS 

TREATMENT  PROCESS  AND  SERVICES  PROVIDED; 

Sununary  of  Requirements: 

The  following  supportive  services  must  be  available: 

-Recreation 

-Rehabilitation  (Vocational) 

-Individual,  Family  and  Group  Counseling 

-Medical  Services  under  Physician  Supervision 

-Emergency  Services  (24  Hours  per  Day) 

-Referral  Services  Including  a  Current  List  of  Resources 

-Follow-Up  Services 

In  addition,  most  programs  are  currently  providing  community 
consultation  and  education  services.   While  these  services  for 
alcoholism  treatment  programs  are  not  required,  evaluation  is 
merited  due  to  the  amount  of  staff  time  spent  in  these  activities. 

Evaluation  Procedures: 


To  assess  whether  all  required  services  are  available,  interview 
the  treatment  coordinator  and  prepare  descriptions  of  the  services 
offered.   For  each  service  program  (inpatient,  outpatient,  resi- 
dential, etc.)  describe  agency  procedures  for  intake,  counselor 
assignment  and  provision  of  supportive  services,  such  as  individual 
and  group  counseling,  vocational  rehabilitation,  etc. 

.Are  all  required  services  available? 

•During  review  of  active  client  case  records  for  the  most 
recent  month  (see  below)  verify  and  document  the  frequency  of 
supportive  service  contact  for  each  service  provided. 
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In  addition,  perform  the  following  steps  for  emergency  services, 
referral  and  follow-up  services,  and  community  consultation  and 
education  activities. 

1 .  Emergency  Services: 

To  assess  the  extent  to  which  emergency  services  are  provided, 
perform  the  following  steps: 

.Obtain  and  review  the  schedule  of  staff  assigned  for  24- 
hour,  7  day  a  week  coverage  for  emergency  services. 

.Is  a  treatment  professional  on  call  for  face-to-face 
contact  at  all  times? 

.Document  the  emergency  services  telephone  number  and 
document  the  method  by  which  it  is  communicated  to  the 
public. 

2 ,  Termination  Procedures  and  Follow-up  and  Referral  Services: 

To  assess  the  extent  to  which  a  system  of  client  termination 
and  follow-up  and  a  system  of  referring  clients  to  other 
service  agencies  have  been  implemented,  perform  the  following 
steps: 
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a.    Describe  the  internal  program  procedures  for  documenting 
client  termination  and  conducting  client  follow-up  and 
referral  after  discharge  from  the  treatment  facility. 
Obtain  access  to  case  records  of  discharged  or  inactive 
clients. 

.To  verify  that  stated  program  procedures  are  actually 
operating,  select  a  sample  of  15  inactive  clients 
discharged  in  the  last  six  months  and  complete  the 
"Termination  and  Follow-up  Documentation  Worksheet", 
Exhibit  IV-15. 

.Do  results  indicate  that  the  program  has  implemented 
a  client  follow-up  system? 

.Are  reasons  for  client  termination  documented? 

.Does  it  appear  that  the  agency  utilizes  other 
service  programs  for  client  referral? 

3.    Community  Consultation  and  Education  Activities: 

To  assess  the  extent  to  which  the  agency  plans  and  monitors 
community  consultation  and  education  activities,  interview  the 
program  director  and  perform  the  following  steps: 
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a.  Describe  those  community  consultation  and  education  (C/E) 
activities  conducted  during  the  past  six  months.  Include 
in  the  description,  the  following: 

1.  The  purposes  or  objectives  of  the  C/E  activities. 

2.  The  names  of  community  groups  or  agencies  contacted. 

3.  The  number  and  type  of  program  staff  conducting  C/E 
activities, 

b.  Obtain  the  most  recent  monthly  report  of  staff  hours 
spent  on  C/E  activities.   Trace  the  hours  reported  back 
to  detailed  supporting  documentation  prepared  by  staff 
such  as  daily  or  weekly  time  cards  or  logs.   Are  reported 
hours  supported  by  detailed  documentation? 

c.  Does  it  appear  that  C/E  activities  are  well  planned  and 
monitored?  Consider  the  following: 

-Can  the  agency  show  evidence  that  objectives  of  C/E 
activities  have  been  achieved? 

-Are  progress  reports  prepared  by  staff  conducting 
C/E  activities?   Are  reports  submitted  to  the 
program  director? 

-Does  the  frequency  and  nature  of  C/E  activities 
justify  the  amount  of  staff  positions  and  time  spent 
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on  C/E  activities?  Document  the  rationale  for  your 
assessment. 


B.    CLIENT  CENSUS  AND  REPORTING: 

Summary  of  Requirements: 

Program  reporting  systems  should  produce  accurate  reports  of: 

-The  number  of  active  clients. 

-Other  data  that  may  be  requested  by  the  Department  of  Insti- 
tutions . 

These  reports  should  be  supported  by  adequate  documentation  to 
validate  report  accuracy. 

Evaluation  Procedures: 


1 .   Process  for  Report  Preparation: 

To  assess  the  process  for  report  preparation  and  methods  of 
documentation,  interview  the  individual(s)  responsible  for 
preparation  of  program  reports  (not  financial)  and  perform  the 
following  steps: 

a.  Prepare  a  brief  flow  chart  of  the  steps  used  for  prepara- 
tion of  the  Monthly  Data  Report.   This  chart  should  begin 
with  treatment  staff  documentation  of  clients  served, 
services  provided  and  staff  hours  spent,  and  end  with 
submittal  of  the  monthly  report  to  the  Department  of 
Institutions. 

b.  Describe  methods  used,  if  any,  by  program  management  to 
validate  information  supplied  by  treatment  staff. 
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NOTE:    If  no  verification  is  conducted,  this  is 
a  weakness  and  should  be  documented. 

c.  Collect  copies  of  the  Monthly  Data  Report  for  the  last 
six  months.   During  the  interview  with  the  individual 
responsible  for  report  preparation,  determine  the  fol- 
lowing: 

.What  criteria  are  used  for  client  admission  and  dis- 
charge for  each  of  the  following  programs: 

— Inpatient 
— Outpatient 
— Residential 

.What  criteria  are  used  for  determination  of  active 
versus  inactive  clients  regarding  frequency  of 
contact? 

d.  Assess  whether  the  system  used  is  reasonable.   Interview 
3-5  professional  staff  to  verify  that  reporting  pro- 
cedures and  definitions  as  described  above  have  been 
communicated  to  staff.   (See  the  "Professional  Staff 
Interview  Questionnaire"  Exhibit  II-2  in  the  Personnel 
Management  and  Staff  Development  Evaluation  Guide.) 
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2.   Accuracy  of  Reports: 

To  assess  the  accuracy  of  the  Monthly  Data  Report,  documenta- 
tion of  the  number  of  outpatients  served  for  the  most  recent 
month  should  be  performed,  as  follows: 

NOTE:  The  client  census  review  should  be  conducted 
simultaneously  with  the  client  record  review 
as  described  in  the  next  section. 

a.   Using  the  individual  client  case  record  or  other  qual- 
ified source,  (medication  dispensing  logs,  group  therapy 
attendance  logs,  etc.),  complete  the  "Outpatient  Client 
Census  Documentation  Worksheet",  Exhibit  IV-16. 

.To  obtain  the  list  of  active  outpatients,  the 
evaluator  should  request  either  a  centralized  list 
of  active  clients  or  access  to  the  active  client 
files.   The  evaluator  should  include  as  active, 
those  clients  discharged  during  the  current  month 
who  were  active  during  the  test  month.   The  eval- 
uator should  find  records  available  for  at  least  the 
number  of  outpatients  reported  on  the  Monthly  Data 
Report . 

.Enter  the  client  case  number  for  each  reported 
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EXHIBIT  IV- 16 

OUTPATIENT 
CLIENT  CENSUS  DO  CUT-EN  TAT  ION  WORKSHEET 


Program  Name : 
Reviewed  BY: 


Service  Program: 


CLIENT   I.D. 

MOST  RECENT     MONIH 
(    ) 
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Documentation  Worksheet",  Exhibit  IV-16,  review 
documentation  supplied  by  the  program  and  enter  a 
check  mark  (■>/)    in  the  column  next  to  the  client 
number  FOR  EACH  CLIENT  FOR  WHOM  AT  LEAST  ONE  FACE-TO- 
FACE  CONTACT  IS  DOCUMENTED  FOR  THE  TEST  MONTH. 

Compare  the  documented  monthly  outpatient  census  to  the 
reported  outpatient  census  for  the  same  month. 

.Complete  the  "Documented  and  Reported  Census  Recon- 
ciliation VJorksheet",  Exhibit  IV-17. 

•For  any  site  or  service  program  with  a  reported 
versus  documented  census  variance  of  more  than  10%, 
review  results  of  analysis  and  probable  causes  of 
the  problem  with  program  management.   Some  possible 
causes  of  the  problem  might  be : 

--Failure  of  treatment  staff  to  terminate  clients 
on  a  timely  basis. 

--Poor  procedures  for  documenting  client  contacts 
in  case  records  or  other  sources. 

•Complete  an  additional  one  month  of  census  docu- 
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mentation  for  programs  and  sites  with  variances  of 
greater  than  10%. 

c.   Calculate  the  average  client  census  as  follows: 

.For  those  programs  and  sites  with  less  than  10% 
variance  in  reported  versus  documented  census,  cal- 
culate the  average  census  for  the  past  six  months 
using  Monthly  Data  Reports. 

.For  those  programs  and  sites  with  more  than  a  10% 
variance,  calculate  the  average  census  for  the  two 
months  reviewed . 

3.    Average  Utilization  of  Residential  and  Inpatient  Beds: 

To  calculate  the  average  utilization  for  residential  and 
inpatient  facilities  perform  the  following  steps: 

a.  Obtain  attendance  records  from  residential  or  inpatient 
facilities  and  determine: 

.Number  of  beds  available  during  the  month 
.Number  of  bed-days  used  during  the  month 

b.  Derive  average  utilization  for  the  month  by  dividing  the 
number  of  bed-days  used  during  the  month,  by  the  number 
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of  bed-days  available  during  the  month.  (Number  of  bed- 
days  available  during  the  month  =  number  of  beds  avail- 
able during  the  month  x  number  of  days  in  the  month.) 

If  the  average  utilization  is  relatively  low  (70%  or 
less)  develop  an  assessment  of  why  the  facilities  are 
underutilized . 

(Consider) 

.Does  the  client  turnover  rate  appear  to  be  exces- 
sive? 


CLIENT  RECORDS: 

Summary  of  Requirements: 

.Active  client  records  must  contain: 

-Individualized  Client  Treatment  Plan 

-Date  of  Admission 

-Physical  Exam  (as  needed) 

-Documentation  of  all  Supportive  Service  Contacts 

-Semi-monthly  Progress  Notes 

-Social  History 

-Medical  History 

.A  resident  admission  register  must  be  maintained. 

In  addition,  client  recui.ds  should  be  well  organized,  uni- 
formly maintained,  and  stored  in  locked  file  cabinets  in 
accordance  with  confidentiality  requirements. 

Evaluation  Procedures: 


The  review  of  client  records  is  to  accomplish  three  primary  ob- 
jectives as  follows: 
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.To  document  the  frequency  and  duration  of  client  treatment 
contact. 

.To  determine  whether  case  records  contain  required  client 
treatment  documentation. 

.To  assess  the  quality  of  case  record  organization  and  main- 
tenaiice  including  record  storage. 

To  meet  these  objectives,  the  evaluator  should  perform  the  fol- 
lowing evaluation  procedures: 

1 .   Methods  of  Recording: 

To  understand  the  program's  record  keeping  system,  interview 
the  program  treatment  coordinator  and  perform  the  following 
steps: 

a.    Obtain  2-3  sample  active  client  records  and  review  each 
record  with  the  treatment  coordinator.   Describe  the 
forms  used  for  the  following: 

.Date  of  Admission 

.Social  History 

.Medical  History 

.Physical  Examinations  (as  needed) 

.Individualized  Treatment  Plan 
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.Plan  Assessment  and  Progress  Notes 
.Documentation  of  Services  provided 
.Resident  Admission  Register 
.Other 

b.    Obtain  a  copy  of  each  form 

Record  Content: 

To  evaluate  content  of  client  records,  obtain  access  to  files 
of  active  client  case  records  and  perform  the  following  steps; 

a.  Select  a  sample  of  case  records  of  presently  active 
clients  not  to  exceed  100  records.   (If  the  program  has 
fewer  than  100  active  clients,  document  record  content 
for  all  clients.) 

NOTE:    For  programs  with  more  than  100  clients, 

randomly  select  client  records  for  review. 
For  example,  for  programs  with  500  clients, 
select  every  fifth  record,  etc. 

b.  Utilizing  the  "Client  Record  Documentation  Worksheet", 
Exhibit  IV-18,  document  record  content  for  each  program 
type  and  site  as  follows: 
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.Client  Number: 

The  evaluator  is  cautioned  to  ensure  client 
confidentiality  by  utilizing  only  the  client 
identification  number.   Names  may  be  used  on- 
site  but  should  be  erased  from  worksheets 
before  leaving. 

•Admission  Date: 

To  calculate  average  client  retention  (see 
below)  enter  the  admission  date  for  each 
client. 

.Content  of  Records: 

Based  on  the  description  of  program  procedures, 
look  for  the  presence  in  each  file  of  the 
indicated  documentation. 

.Frequency  of  Supportive  Services  Contact: 

Record  the  number  of  documented  face-to-face 
contacts  by  type  of  service  for  the  most  recent 
month.   Documentation  should  consist  of  written 
counseling  notes  or  other  qualified  sources 
such  as  group  counseling  attendance  records. 
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The  evaluator  should  request  the  assistance  of 
program  staff  to  locate  all  reasonably  acces- 
sible documentation  of  contact.   Family  coun- 
seling contacts  need  not  include  the  primary 
client  as  a  participant. 

NOTE:     To  help  ensure  that  clients  are 
receiving  necessary  services  and 
that  staff  time  is  efficiently 
utilized,  all  supportive  service 
contacts  should  be  documented  in 
case  records  as  they  occur, 
e.g.,  case  files  for  residential 
clients  should  include  docximenta- 
tion  of  individual  service 
contacts,  not  just  documentation 
of  service  days  received.   If 
individual  contacts  are  not 
documented,  this  is  a  weakness 
and  should  be  noted. 

To  assess  the  extent  to  which  progress  notes  in  client 
records  are  consistent  with  individualized  treatment 
plans  and  the  extent  to  which  the  updating  of  these 
treatment  plans  reflects  progress  which  has  been  re- 
ported, perform  the  following  steps: 
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.Select  a  sample  of  (15)  client  case  records  from  the 
"Client  Record  Documentation  Worksheet",  Exhibit  IV- 
18,  and  perform  the  following  steps: 

.Review  the  individualized  treatment  plan  and  progress 
notes  for  each  case  record.   Determine  whether  they 
are  consistent  with  one  another,  i.e.,  whether 
progress  toward  the  specific  objectives  in  the 
treatment  plan  is  mentioned  in  the  notes,  and  whether 
treatment  staff  recommendations  become  part  of  the 
plan  objectives  which  are  mentioned  in  subsequent 
progress  notes. 

.Are  progress  notes  consistent  with  treatment  plans? 
.Have  there  been  at  least  semi-monthly  progress 
notes? 

d.   Summarize  results  of  the  record  documentation  review  by 
calculating  the  following  statistics: 

.Client  Retention: 

Based  on  the  date  of  admission,  determine  the 
number  and  percent  of  clients  who  have  been 
active  for : 
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0-1  Month  (less  than  30  days) 
1-3  Months  (30-90  days) 
3-6  Months 
6-+  Months 

.Record  Content: 

Calculate  the  number  and  percent  of  client 
records  lacking  the  required  documentation. 

.Frequency  of  Supportive  Services  Contact  by  Client: 

Calculate  the  number  and  percent  of  clients 
provided  services  the  following  number  of  times 
during  the  most  recent  month: 


0 

1-3 

4-6 

7-9 

10-+ 


.Frequency  of  Supportive  Services  contact  by  admission 
date  of  client: 

Calculate  the  number  and  percent  of  clients  in 
each  of  the  retention  categories  provided 
services  in  each  of  the  contact  of  frequency 
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CATEGORIES.   Complete  the  "Frequency  of  Sup- 
portive Services  Contact  By  Admission  Date  of 
Client",  Worksheet  Exhibit  IV-19. 

.Frequency  of  Supportive  Service  contact  by  service: 

Calculate  the  total  number  and  percent  of 
contacts  provided  to  all  clients  by  service 
during  the  most  recent  month,  as  follows; 

TYPE  OF  SERVICE      NO.  OF  CONTACTS      %  OF  TOTAL  CONTACTS 


Individual 
Counseling 

Group  Counseling 

Family  Counseling, 
etc . 

TOTAL  100% 


e.   Analyze  results  of  the  record  documentation  review. 
Consider  the  following: 

.How  does  the  program  compare  with  other  programs  in 
client  retention? 

.Is  required  content  present  for  at  least  90%  of  all 
records  reviewed?   If  not,  discuss  reasons  with 
program  management. 


< 
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•Do  records  accurately  reflect  the  extent  and  nature 
of  supportive  services  provided?   How  does  the 
program  compare  with  other  programs  in  frequency 
of  contact? 

.Does  it  appear  that  newer  clients  have  a  greater 
frequency  of  service  contact? 

.Are  overall  service  contacts  by  type  reasonably 
distributed,  or  does  it  appear  that  only  one  or  two 
types  of  services  are  being  provided  and  documented? 

.Are  records  stored  in  locked  file  cabinets?   Do 
records  appear  well  organized  and  uniformly  main- 
tained? 

.Review  all  findings  with  program  management  and 
identify  reasons  for  problems  observed  and  solutions 
program  management  intends  to  implement. 

NOTE:     These  solutions  should  become  part 

of  the  program's  treatment  obiectives 
for  the  next  program  year. 
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APPENDIX  A 


PROGRAM  EVALUATION  CHECKLIST 
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